2023-07-20T18:37:02.873 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T18:37:02.874 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --archive /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417 --name pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/orig.config.yaml 2023-07-20T18:37:02.934 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T18:37:03.037 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: pdonnell@redhat.com first_in_suite: false job_id: '7345417' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 78296bf9274b016ce68ff036026e51a56af22f7c owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8388 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 12/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 78296bf9274b016ce68ff036026e51a56af22f7c syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.01 - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_16:47:59 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T18:37:03.037 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa; will attempt to use it 2023-07-20T18:37:03.038 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks 2023-07-20T18:37:03.038 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T18:37:03.040 INFO:teuthology.task.internal:Checking packages... 2023-07-20T18:37:03.071 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T18:37:03.072 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T18:37:03.072 INFO:teuthology.packaging:ref: None 2023-07-20T18:37:03.072 INFO:teuthology.packaging:tag: None 2023-07-20T18:37:03.072 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T18:37:03.073 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:37:03.073 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T18:37:03.383 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T18:37:03.384 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T18:37:03.391 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T18:37:03.392 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T18:37:03.402 INFO:teuthology.task.internal:Saving configuration 2023-07-20T18:37:03.435 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T18:37:03.441 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T18:37:03.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:32:04.780941', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T18:37:03.498 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:32:04.779819', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T18:37:03.521 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:32:04.778471', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T18:37:03.521 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T18:37:03.527 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T18:37:03.528 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T18:37:03.528 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T18:37:03.528 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T18:37:03.650 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8218eca430>, signals=[15]) 2023-07-20T18:37:03.652 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T18:37:03.661 INFO:teuthology.task.internal:Opening connections... 2023-07-20T18:37:03.661 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2023-07-20T18:37:03.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:37:03.737 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2023-07-20T18:37:03.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:37:03.811 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2023-07-20T18:37:03.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:37:03.881 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T18:37:03.892 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2023-07-20T18:37:03.915 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2023-07-20T18:37:03.915 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2023-07-20T18:37:03.975 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T18:37:03.975 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T18:37:03.975 INFO:teuthology.orchestra.run.smithi061.stdout:ID="rhel" 2023-07-20T18:37:03.976 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="fedora" 2023-07-20T18:37:03.976 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8.6" 2023-07-20T18:37:03.976 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2023-07-20T18:37:03.976 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T18:37:03.976 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2023-07-20T18:37:03.977 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T18:37:03.977 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T18:37:03.977 INFO:teuthology.orchestra.run.smithi061.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T18:37:03.977 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T18:37:03.977 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:37:03.977 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T18:37:03.978 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T18:37:03.978 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T18:37:03.978 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T18:37:03.979 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-07-20T18:37:04.005 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2023-07-20T18:37:04.027 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2023-07-20T18:37:04.027 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2023-07-20T18:37:04.088 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T18:37:04.088 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T18:37:04.089 INFO:teuthology.orchestra.run.smithi149.stdout:ID="rhel" 2023-07-20T18:37:04.089 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="fedora" 2023-07-20T18:37:04.089 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8.6" 2023-07-20T18:37:04.089 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2023-07-20T18:37:04.090 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T18:37:04.090 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2023-07-20T18:37:04.090 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T18:37:04.090 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T18:37:04.091 INFO:teuthology.orchestra.run.smithi149.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T18:37:04.091 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T18:37:04.091 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:37:04.091 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T18:37:04.092 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T18:37:04.092 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T18:37:04.092 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T18:37:04.093 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2023-07-20T18:37:04.113 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2023-07-20T18:37:04.133 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2023-07-20T18:37:04.133 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2023-07-20T18:37:04.193 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T18:37:04.193 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T18:37:04.193 INFO:teuthology.orchestra.run.smithi156.stdout:ID="rhel" 2023-07-20T18:37:04.194 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="fedora" 2023-07-20T18:37:04.194 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8.6" 2023-07-20T18:37:04.194 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8" 2023-07-20T18:37:04.194 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T18:37:04.195 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2023-07-20T18:37:04.195 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T18:37:04.195 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T18:37:04.195 INFO:teuthology.orchestra.run.smithi156.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T18:37:04.195 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T18:37:04.196 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:37:04.196 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T18:37:04.196 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T18:37:04.196 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T18:37:04.196 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T18:37:04.197 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-07-20T18:37:04.222 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T18:37:04.234 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T18:37:04.243 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T18:37:04.243 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2023-07-20T18:37:04.246 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2023-07-20T18:37:04.248 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2023-07-20T18:37:04.266 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T18:37:04.273 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T18:37:04.274 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2023-07-20T18:37:04.306 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2023-07-20T18:37:04.308 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2023-07-20T18:37:04.376 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T18:37:04.468 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T18:37:04.491 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-20T18:37:04.492 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-20T18:37:04.492 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-20T18:37:04.493 DEBUG:teuthology.orchestra.run.smithi061:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T18:37:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:37:04.513 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-07-20T18:37:04.572 INFO:teuthology.orchestra.run.smithi061.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T18:37:04.573 INFO:teuthology.task.kernel:Running kernel on smithi061: 4.18.0-372.9.1.el8.x86_64 2023-07-20T18:37:04.573 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum install -y kernel 2023-07-20T18:37:05.829 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:37:05.829 INFO:teuthology.orchestra.run.smithi061.stdout:Unable to read consumer identity 2023-07-20T18:37:07.319 INFO:teuthology.orchestra.run.smithi061.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-07-20T18:37:08.103 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-07-20T18:37:11.753 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 438 kB/s | 24 kB 00:00 2023-07-20T18:37:12.439 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T18:37:12.488 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-07-20T18:37:12.488 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-07-20T18:37:12.489 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-07-20T18:37:12.616 DEBUG:teuthology.orchestra.run.smithi061:> echo no | sudo yum reinstall kernel || true 2023-07-20T18:37:12.967 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:37:12.968 INFO:teuthology.orchestra.run.smithi061.stdout:Unable to read consumer identity 2023-07-20T18:37:13.250 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 06:37:11 PM UTC. 2023-07-20T18:37:13.296 INFO:teuthology.orchestra.run.smithi061.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T18:37:13.334 INFO:teuthology.orchestra.run.smithi061.stderr:Error: No packages marked for reinstall. 2023-07-20T18:37:13.382 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum reinstall -y kernel || true 2023-07-20T18:37:13.731 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:37:13.731 INFO:teuthology.orchestra.run.smithi061.stdout:Unable to read consumer identity 2023-07-20T18:37:14.008 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:00:03 ago on Thu 20 Jul 2023 06:37:11 PM UTC. 2023-07-20T18:37:14.050 INFO:teuthology.orchestra.run.smithi061.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T18:37:14.078 INFO:teuthology.orchestra.run.smithi061.stderr:Error: No packages marked for reinstall. 2023-07-20T18:37:14.127 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T18:37:14.210 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T18:37:14.211 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T18:37:14.211 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T18:37:14.211 DEBUG:teuthology.parallel:result is None 2023-07-20T18:37:14.211 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T18:37:14.218 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T18:37:14.219 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T18:37:14.221 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T18:37:14.223 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T18:37:14.245 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T18:37:14.252 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T18:37:14.259 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T18:37:14.260 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T18:37:14.278 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T18:37:14.286 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T18:37:14.437 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T18:37:14.444 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T18:37:14.444 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T18:37:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:37:14.464 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T18:37:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:37:14.484 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T18:37:14.487 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T18:37:14.508 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T18:37:14.528 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:37:14.541 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:37:14.559 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:37:14.571 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:37:14.577 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:37:14.589 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T18:37:14.590 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T18:37:14.601 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T18:37:14.602 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T18:37:14.604 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T18:37:14.606 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T18:37:14.635 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T18:37:14.643 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T18:37:14.644 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T18:37:14.672 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T18:37:14.676 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T18:37:14.693 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:37:14.775 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:37:14.827 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:37:14.914 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:37:14.953 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:37:14.953 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T18:37:15.027 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:37:15.079 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:37:15.137 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:37:15.192 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:37:15.232 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T18:37:15.233 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T18:37:15.301 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:37:15.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T18:37:15.399 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:37:15.486 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T18:37:15.525 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T18:37:15.526 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T18:37:15.594 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2023-07-20T18:37:15.598 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2023-07-20T18:37:15.600 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2023-07-20T18:37:15.653 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T18:37:15.657 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T18:37:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T18:37:16.004 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T18:37:16.013 INFO:teuthology.task.internal:Starting timer... 2023-07-20T18:37:16.013 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T18:37:16.022 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T18:37:16.031 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T18:37:16.032 DEBUG:teuthology.orchestra.run.smithi061:> sudo service auditd rotate 2023-07-20T18:37:16.091 INFO:teuthology.orchestra.run.smithi061.stdout:Rotating logs: 2023-07-20T18:37:16.093 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2023-07-20T18:37:16.157 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2023-07-20T18:37:16.159 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2023-07-20T18:37:16.218 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2023-07-20T18:37:16.220 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T18:37:16.220 DEBUG:teuthology.orchestra.run.smithi061:> /usr/sbin/getenforce 2023-07-20T18:37:16.249 INFO:teuthology.orchestra.run.smithi061.stdout:Permissive 2023-07-20T18:37:16.250 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2023-07-20T18:37:16.279 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2023-07-20T18:37:16.280 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2023-07-20T18:37:16.301 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2023-07-20T18:37:16.301 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi061.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2023-07-20T18:37:16.302 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T18:37:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:37:16.332 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T18:37:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:37:16.360 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T18:37:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T18:37:16.388 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T18:37:16.388 DEBUG:teuthology.orchestra.run.smithi061:> sudo /usr/sbin/setenforce permissive 2023-07-20T18:37:16.417 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2023-07-20T18:37:16.449 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2023-07-20T18:37:16.493 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T18:37:16.502 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-20T18:37:16.503 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T18:37:16.524 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T18:37:16.525 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com,smithi149.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T18:54:56.595 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2023-07-20T18:54:56.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-07-20T18:54:56.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:54:56.682 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-07-20T18:54:56.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-07-20T18:54:56.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-07-20T18:54:56.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:54:56.824 DEBUG:teuthology.orchestra.run.smithi149:> true 2023-07-20T18:54:56.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-07-20T18:54:56.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-07-20T18:54:56.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T18:54:56.960 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-07-20T18:54:56.984 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-07-20T18:54:56.985 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T18:54:56.996 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T18:54:56.997 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T18:54:56.997 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T18:54:57.000 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T18:54:57.001 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T18:54:57.002 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T18:54:57.003 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T18:54:57.032 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T18:54:57.033 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T18:54:57.046 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T18:54:57.048 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T18:54:57.056 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T18:54:57.072 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T18:54:57.120 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2023-07-20T18:54:57.121 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: ntpd: command not found 2023-07-20T18:54:57.132 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2023-07-20T18:54:57.133 INFO:teuthology.orchestra.run.smithi061.stdout:506 Cannot talk to daemon 2023-07-20T18:54:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2023-07-20T18:54:57.146 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T18:54:57.148 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2023-07-20T18:54:57.149 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T18:54:57.160 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T18:54:57.163 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T18:54:57.164 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T18:54:57.176 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T18:54:57.240 INFO:teuthology.orchestra.run.smithi061.stderr:bash: ntpq: command not found 2023-07-20T18:54:57.241 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2023-07-20T18:54:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T18:54:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2023-07-20T18:54:57.242 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.243 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.244 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.244 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.249 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-07-20T18:54:57.252 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T18:54:57.253 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-07-20T18:54:57.253 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.253 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.253 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.253 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T18:54:57.254 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T18:54:57.264 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T18:54:57.269 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi061.front.sepia.ceph.com 2023-07-20T18:54:57.269 DEBUG:teuthology.orchestra.run.smithi061:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T18:54:57.270 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi149.front.sepia.ceph.com 2023-07-20T18:54:57.270 DEBUG:teuthology.orchestra.run.smithi149:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T18:54:57.270 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi156.front.sepia.ceph.com 2023-07-20T18:54:57.271 DEBUG:teuthology.orchestra.run.smithi156:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T18:54:57.660 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-07-20T18:54:57.666 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-20T18:54:57.676 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:54:59.084 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:06:33 ago on Thu 20 Jul 2023 06:48:25 PM UTC. 2023-07-20T18:54:59.305 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:18 ago on Thu 20 Jul 2023 06:52:40 PM UTC. 2023-07-20T18:54:59.373 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:08:41 ago on Thu 20 Jul 2023 06:46:17 PM UTC. 2023-07-20T18:55:00.824 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-07-20T18:55:00.825 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-07-20T18:55:00.825 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout:Resetting modules: 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-07-20T18:55:00.826 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:01.070 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-07-20T18:55:01.071 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-07-20T18:55:01.071 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2023-07-20T18:55:01.071 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-07-20T18:55:01.071 INFO:teuthology.orchestra.run.smithi149.stdout:Resetting modules: 2023-07-20T18:55:01.071 INFO:teuthology.orchestra.run.smithi149.stdout: container-tools 2023-07-20T18:55:01.072 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:01.072 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-07-20T18:55:01.072 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2023-07-20T18:55:01.072 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:01.371 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-07-20T18:55:01.372 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-07-20T18:55:01.372 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout:Resetting modules: 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-07-20T18:55:01.373 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:01.789 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:06:36 ago on Thu 20 Jul 2023 06:48:25 PM UTC. 2023-07-20T18:55:02.059 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:21 ago on Thu 20 Jul 2023 06:52:40 PM UTC. 2023-07-20T18:55:02.374 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:08:44 ago on Thu 20 Jul 2023 06:46:17 PM UTC. 2023-07-20T18:55:08.723 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-07-20T18:55:09.207 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-07-20T18:55:10.923 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:30 ago on Thu 20 Jul 2023 06:52:40 PM UTC. 2023-07-20T18:55:11.069 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-07-20T18:55:11.554 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-20T18:55:12.598 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-07-20T18:55:12.641 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-07-20T18:55:12.735 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================================ 2023-07-20T18:55:12.735 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-07-20T18:55:12.735 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================================ 2023-07-20T18:55:12.735 INFO:teuthology.orchestra.run.smithi149.stdout:Installing group/module packages: 2023-07-20T18:55:12.735 INFO:teuthology.orchestra.run.smithi149.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T18:55:12.735 INFO:teuthology.orchestra.run.smithi149.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T18:55:12.736 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout:Installing module profiles: 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout: container-tools/common 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout:Enabling module streams: 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout: container-tools rhel8 2023-07-20T18:55:12.737 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:12.738 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-07-20T18:55:12.738 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================================ 2023-07-20T18:55:12.738 INFO:teuthology.orchestra.run.smithi149.stdout:Install 10 Packages 2023-07-20T18:55:12.738 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:12.738 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 18 M 2023-07-20T18:55:12.739 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 65 M 2023-07-20T18:55:12.739 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-07-20T18:55:12.842 INFO:teuthology.orchestra.run.smithi149.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 897 kB/s | 94 kB 00:00 2023-07-20T18:55:12.876 INFO:teuthology.orchestra.run.smithi149.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 295 kB/s | 41 kB 00:00 2023-07-20T18:55:12.901 INFO:teuthology.orchestra.run.smithi149.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 152 kB/s | 25 kB 00:00 2023-07-20T18:55:12.943 INFO:teuthology.orchestra.run.smithi149.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 492 kB/s | 49 kB 00:00 2023-07-20T18:55:13.018 INFO:teuthology.orchestra.run.smithi149.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.6 MB/s | 524 kB 00:00 2023-07-20T18:55:13.025 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:06:47 ago on Thu 20 Jul 2023 06:48:25 PM UTC. 2023-07-20T18:55:13.044 INFO:teuthology.orchestra.run.smithi149.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.5 MB/s | 149 kB 00:00 2023-07-20T18:55:13.068 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:55:13.118 INFO:teuthology.orchestra.run.smithi149.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.1 MB/s | 213 kB 00:00 2023-07-20T18:55:13.227 INFO:teuthology.orchestra.run.smithi149.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-07-20T18:55:13.386 INFO:teuthology.orchestra.run.smithi149.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 21 MB/s | 7.0 MB 00:00 2023-07-20T18:55:13.444 INFO:teuthology.orchestra.run.smithi149.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 6.7 MB/s | 2.2 MB 00:00 2023-07-20T18:55:13.445 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:55:13.445 INFO:teuthology.orchestra.run.smithi149.stdout:Total 26 MB/s | 18 MB 00:00 2023-07-20T18:55:13.645 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-07-20T18:55:13.664 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-07-20T18:55:13.664 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-07-20T18:55:13.846 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-07-20T18:55:13.850 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-07-20T18:55:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-07-20T18:55:14.553 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T18:55:14.689 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:08:57 ago on Thu 20 Jul 2023 06:46:17 PM UTC. 2023-07-20T18:55:14.713 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:55:14.753 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-07-20T18:55:14.845 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout:Installing group/module packages: 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T18:55:14.846 INFO:teuthology.orchestra.run.smithi156.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T18:55:14.847 INFO:teuthology.orchestra.run.smithi156.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout:Installing module profiles: 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools/common 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling module streams: 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools rhel8 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-07-20T18:55:14.848 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-07-20T18:55:14.849 INFO:teuthology.orchestra.run.smithi156.stdout:Install 10 Packages 2023-07-20T18:55:14.849 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:14.849 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 18 M 2023-07-20T18:55:14.849 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 65 M 2023-07-20T18:55:14.850 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-07-20T18:55:14.923 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T18:55:15.009 INFO:teuthology.orchestra.run.smithi156.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 582 kB/s | 94 kB 00:00 2023-07-20T18:55:15.034 INFO:teuthology.orchestra.run.smithi156.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 133 kB/s | 25 kB 00:00 2023-07-20T18:55:15.068 INFO:teuthology.orchestra.run.smithi156.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 185 kB/s | 41 kB 00:00 2023-07-20T18:55:15.108 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T18:55:15.135 INFO:teuthology.orchestra.run.smithi156.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 393 kB/s | 49 kB 00:00 2023-07-20T18:55:15.193 INFO:teuthology.orchestra.run.smithi156.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.2 MB/s | 524 kB 00:00 2023-07-20T18:55:15.352 INFO:teuthology.orchestra.run.smithi156.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 28 MB/s | 8.0 MB 00:00 2023-07-20T18:55:15.378 INFO:teuthology.orchestra.run.smithi156.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 617 kB/s | 149 kB 00:00 2023-07-20T18:55:15.458 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T18:55:15.469 INFO:teuthology.orchestra.run.smithi156.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 773 kB/s | 213 kB 00:00 2023-07-20T18:55:15.628 INFO:teuthology.orchestra.run.smithi156.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 8.8 MB/s | 2.2 MB 00:00 2023-07-20T18:55:15.720 INFO:teuthology.orchestra.run.smithi156.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 19 MB/s | 7.0 MB 00:00 2023-07-20T18:55:15.721 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:55:15.721 INFO:teuthology.orchestra.run.smithi156.stdout:Total 21 MB/s | 18 MB 00:00 2023-07-20T18:55:15.922 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-07-20T18:55:15.940 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-07-20T18:55:15.941 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-07-20T18:55:16.123 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-07-20T18:55:16.127 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-07-20T18:55:16.228 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T18:55:16.661 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-07-20T18:55:16.666 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-07-20T18:55:16.762 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================================ 2023-07-20T18:55:16.762 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================================ 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout:Installing group/module packages: 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T18:55:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T18:55:16.764 INFO:teuthology.orchestra.run.smithi061.stdout:Installing module profiles: 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools/common 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling module streams: 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools rhel8 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================================ 2023-07-20T18:55:16.765 INFO:teuthology.orchestra.run.smithi061.stdout:Install 10 Packages 2023-07-20T18:55:16.766 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:16.766 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 18 M 2023-07-20T18:55:16.766 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 65 M 2023-07-20T18:55:16.767 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-07-20T18:55:16.872 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T18:55:16.908 INFO:teuthology.orchestra.run.smithi061.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 174 kB/s | 25 kB 00:00 2023-07-20T18:55:16.933 INFO:teuthology.orchestra.run.smithi061.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 243 kB/s | 41 kB 00:00 2023-07-20T18:55:16.983 INFO:teuthology.orchestra.run.smithi061.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 430 kB/s | 94 kB 00:00 2023-07-20T18:55:17.040 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:55:17.067 INFO:teuthology.orchestra.run.smithi061.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 311 kB/s | 49 kB 00:00 2023-07-20T18:55:17.125 INFO:teuthology.orchestra.run.smithi061.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.7 MB/s | 524 kB 00:00 2023-07-20T18:55:17.137 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T18:55:17.226 INFO:teuthology.orchestra.run.smithi061.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 942 kB/s | 149 kB 00:00 2023-07-20T18:55:17.226 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T18:55:17.259 INFO:teuthology.orchestra.run.smithi061.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.6 MB/s | 213 kB 00:00 2023-07-20T18:55:17.368 INFO:teuthology.orchestra.run.smithi061.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 21 MB/s | 8.0 MB 00:00 2023-07-20T18:55:17.370 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T18:55:17.430 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T18:55:17.435 INFO:teuthology.orchestra.run.smithi061.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 13 MB/s | 2.2 MB 00:00 2023-07-20T18:55:17.562 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T18:55:17.577 INFO:teuthology.orchestra.run.smithi061.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-07-20T18:55:17.578 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:55:17.578 INFO:teuthology.orchestra.run.smithi061.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-20T18:55:17.722 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:55:17.785 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-07-20T18:55:17.786 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T18:55:17.806 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-07-20T18:55:17.806 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-07-20T18:55:17.997 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:55:17.997 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T18:55:17.997 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:55:17.998 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T18:55:17.998 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T18:55:17.998 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T18:55:17.998 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T18:55:17.999 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T18:55:17.999 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T18:55:17.999 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T18:55:18.001 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-07-20T18:55:18.005 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-07-20T18:55:18.505 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-07-20T18:55:18.533 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T18:55:18.711 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T18:55:18.886 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:55:19.091 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T18:55:19.172 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T18:55:19.172 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:38 ago on Thu 20 Jul 2023 06:52:40 PM UTC. 2023-07-20T18:55:19.333 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T18:55:19.450 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T18:55:19.705 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T18:55:19.726 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T18:55:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T18:55:20.094 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:55:20.369 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:55:20.369 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T18:55:20.369 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:55:20.369 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T18:55:20.369 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T18:55:20.370 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T18:55:20.370 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T18:55:20.370 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T18:55:20.370 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T18:55:20.370 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T18:55:20.559 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T18:55:21.472 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T18:55:21.510 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T18:55:21.510 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:06:55 ago on Thu 20 Jul 2023 06:48:25 PM UTC. 2023-07-20T18:55:21.690 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T18:55:21.851 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T18:55:22.055 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:55:22.330 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T18:55:22.331 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T18:55:22.332 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T18:55:22.332 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T18:55:23.467 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T18:55:23.467 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:09:05 ago on Thu 20 Jul 2023 06:46:17 PM UTC. 2023-07-20T18:55:29.185 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-07-20T18:55:29.391 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:29.391 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-07-20T18:55:29.391 INFO:teuthology.orchestra.run.smithi149.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:29.391 INFO:teuthology.orchestra.run.smithi149.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:55:29.393 INFO:teuthology.orchestra.run.smithi149.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T18:55:29.393 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:55:29.393 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-07-20T18:55:29.559 DEBUG:teuthology.parallel:result is None 2023-07-20T18:55:30.541 INFO:teuthology.orchestra.run.smithi156.stdout:Installed products updated. 2023-07-20T18:55:30.761 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:30.762 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-07-20T18:55:30.762 INFO:teuthology.orchestra.run.smithi156.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:30.762 INFO:teuthology.orchestra.run.smithi156.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:55:30.762 INFO:teuthology.orchestra.run.smithi156.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:30.762 INFO:teuthology.orchestra.run.smithi156.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:55:30.763 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-07-20T18:55:30.940 DEBUG:teuthology.parallel:result is None 2023-07-20T18:55:32.676 INFO:teuthology.orchestra.run.smithi061.stdout:Installed products updated. 2023-07-20T18:55:32.885 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T18:55:32.886 INFO:teuthology.orchestra.run.smithi061.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:32.887 INFO:teuthology.orchestra.run.smithi061.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T18:55:32.887 INFO:teuthology.orchestra.run.smithi061.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T18:55:32.887 INFO:teuthology.orchestra.run.smithi061.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T18:55:32.887 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:55:32.887 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-07-20T18:55:33.070 DEBUG:teuthology.parallel:result is None 2023-07-20T18:55:33.071 INFO:teuthology.run_tasks:Running task install... 2023-07-20T18:55:33.080 DEBUG:teuthology.task.install:project ceph 2023-07-20T18:55:33.081 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T18:55:33.081 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T18:55:33.081 INFO:teuthology.task.install:Using flavor: default 2023-07-20T18:55:33.087 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T18:55:33.087 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T18:55:33.088 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T18:55:33.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:55:33.091 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T18:55:33.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:55:33.092 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T18:55:33.092 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:55:33.418 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T18:55:33.419 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T18:55:38.429 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T18:55:38.429 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T18:55:38.431 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T18:55:38.431 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T18:55:58.610 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T18:55:58.610 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:55:58.611 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T18:55:58.613 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T18:55:58.613 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T18:55:58.614 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T18:55:58.639 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T18:55:58.640 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T18:55:58.640 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T18:55:58.649 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T18:55:58.649 DEBUG:teuthology.orchestra.run.smithi149:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T18:55:58.650 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T18:55:58.651 DEBUG:teuthology.orchestra.run.smithi061:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T18:55:58.675 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T18:55:58.676 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T18:55:58.719 DEBUG:teuthology.orchestra.run.smithi149:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T18:55:58.722 DEBUG:teuthology.orchestra.run.smithi061:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T18:55:58.746 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T18:55:58.799 DEBUG:teuthology.orchestra.run.smithi149:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T18:55:58.803 DEBUG:teuthology.orchestra.run.smithi061:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T18:55:58.825 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T18:55:58.866 INFO:teuthology.orchestra.run.smithi149.stdout:check_obsoletes = 1 2023-07-20T18:55:58.867 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum clean all 2023-07-20T18:55:58.872 INFO:teuthology.orchestra.run.smithi061.stdout:check_obsoletes = 1 2023-07-20T18:55:58.873 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum clean all 2023-07-20T18:55:58.893 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2023-07-20T18:55:58.894 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2023-07-20T18:55:59.241 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-07-20T18:55:59.268 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-20T18:55:59.268 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:56:00.173 INFO:teuthology.orchestra.run.smithi061.stdout:44 files removed 2023-07-20T18:56:00.174 INFO:teuthology.orchestra.run.smithi149.stdout:44 files removed 2023-07-20T18:56:00.225 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T18:56:00.226 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T18:56:00.260 INFO:teuthology.orchestra.run.smithi156.stdout:44 files removed 2023-07-20T18:56:00.312 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T18:56:00.561 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-07-20T18:56:00.579 INFO:teuthology.orchestra.run.smithi061.stdout:Updating Subscription Management repositories. 2023-07-20T18:56:00.647 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-20T18:56:02.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph packages for x86_64 160 kB/s | 81 kB 00:00 2023-07-20T18:56:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 156 kB/s | 81 kB 00:00 2023-07-20T18:56:02.275 INFO:teuthology.orchestra.run.smithi149.stdout:ceph packages for x86_64 173 kB/s | 81 kB 00:00 2023-07-20T18:56:02.679 INFO:teuthology.orchestra.run.smithi061.stdout:ceph noarch packages 21 kB/s | 9.4 kB 00:00 2023-07-20T18:56:02.778 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 20 kB/s | 9.4 kB 00:00 2023-07-20T18:56:02.827 INFO:teuthology.orchestra.run.smithi149.stdout:ceph noarch packages 18 kB/s | 9.4 kB 00:00 2023-07-20T18:56:03.304 INFO:teuthology.orchestra.run.smithi061.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2023-07-20T18:56:03.429 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 2.4 kB/s | 1.5 kB 00:00 2023-07-20T18:56:03.469 INFO:teuthology.orchestra.run.smithi149.stdout:ceph source packages 2.4 kB/s | 1.5 kB 00:00 2023-07-20T18:56:03.671 INFO:teuthology.orchestra.run.smithi061.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-20T18:56:03.713 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-07-20T18:56:03.786 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2023-07-20T18:56:05.314 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2023-07-20T18:56:06.239 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 16 MB 00:02 2023-07-20T18:56:08.022 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 16 MB 00:04 2023-07-20T18:56:08.722 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 378 kB/s | 24 kB 00:00 2023-07-20T18:56:09.841 INFO:teuthology.orchestra.run.smithi156.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 53 MB 00:01 2023-07-20T18:56:09.906 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 369 kB/s | 24 kB 00:00 2023-07-20T18:56:11.149 INFO:teuthology.orchestra.run.smithi061.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-07-20T18:56:11.628 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 164 kB/s | 24 kB 00:00 2023-07-20T18:56:12.780 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 53 MB 00:01 2023-07-20T18:56:18.067 INFO:teuthology.orchestra.run.smithi156.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 47 MB 00:00 2023-07-20T18:56:20.000 INFO:teuthology.orchestra.run.smithi061.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-07-20T18:56:21.490 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 MB/s | 47 MB 00:01 2023-07-20T18:56:29.124 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:56:29.125 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:56:29.139 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T18:56:29.140 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T18:56:29.141 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T18:56:29.142 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T18:56:29.144 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T18:56:29.147 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T18:56:29.149 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T18:56:29.282 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-07-20T18:56:29.295 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-07-20T18:56:29.295 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-07-20T18:56:29.295 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-07-20T18:56:29.296 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-07-20T18:56:29.296 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T18:56:29.296 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T18:56:29.296 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T18:56:29.296 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T18:56:29.296 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T18:56:29.297 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T18:56:29.297 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T18:56:29.298 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T18:56:29.298 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T18:56:29.298 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T18:56:29.298 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T18:56:29.298 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T18:56:29.298 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T18:56:29.299 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T18:56:29.300 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T18:56:29.300 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:56:29.300 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T18:56:29.300 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T18:56:29.301 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T18:56:29.301 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T18:56:29.301 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T18:56:29.301 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T18:56:29.302 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T18:56:29.302 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T18:56:29.302 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T18:56:29.302 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T18:56:29.303 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T18:56:29.303 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T18:56:29.303 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T18:56:29.304 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T18:56:29.304 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2023-07-20T18:56:29.304 INFO:teuthology.orchestra.run.smithi156.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T18:56:29.304 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T18:56:29.305 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T18:56:29.305 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-07-20T18:56:29.305 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T18:56:29.306 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T18:56:29.306 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T18:56:29.306 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T18:56:29.307 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T18:56:29.307 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T18:56:29.307 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T18:56:29.307 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T18:56:29.308 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T18:56:29.308 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T18:56:29.308 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T18:56:29.309 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T18:56:29.309 INFO:teuthology.orchestra.run.smithi156.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T18:56:29.309 INFO:teuthology.orchestra.run.smithi156.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T18:56:29.309 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:56:29.310 INFO:teuthology.orchestra.run.smithi156.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T18:56:29.310 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T18:56:29.310 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T18:56:29.310 INFO:teuthology.orchestra.run.smithi156.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T18:56:29.310 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T18:56:29.311 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T18:56:29.311 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T18:56:29.311 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T18:56:29.312 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T18:56:29.312 INFO:teuthology.orchestra.run.smithi156.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T18:56:29.312 INFO:teuthology.orchestra.run.smithi156.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T18:56:29.312 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T18:56:29.312 INFO:teuthology.orchestra.run.smithi156.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T18:56:29.313 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T18:56:29.314 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T18:56:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T18:56:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T18:56:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T18:56:29.318 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T18:56:29.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T18:56:29.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T18:56:29.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T18:56:29.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T18:56:29.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T18:56:29.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T18:56:29.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T18:56:29.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T18:56:29.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T18:56:29.322 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T18:56:29.322 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T18:56:29.322 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T18:56:29.322 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T18:56:29.323 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T18:56:29.323 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling module streams: 2023-07-20T18:56:29.323 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql 10 2023-07-20T18:56:29.324 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:56:29.324 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-07-20T18:56:29.324 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-07-20T18:56:29.324 INFO:teuthology.orchestra.run.smithi156.stdout:Install 128 Packages 2023-07-20T18:56:29.325 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 3 Packages 2023-07-20T18:56:29.325 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:56:29.326 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 206 M 2023-07-20T18:56:29.326 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-07-20T18:56:29.715 INFO:teuthology.orchestra.run.smithi156.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 29 kB/s | 6.4 kB 00:00 2023-07-20T18:56:30.085 INFO:teuthology.orchestra.run.smithi156.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 2.2 MB/s | 850 kB 00:00 2023-07-20T18:56:30.219 INFO:teuthology.orchestra.run.smithi156.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.2 MB/s | 167 kB 00:00 2023-07-20T18:56:30.710 INFO:teuthology.orchestra.run.smithi156.stdout:(4/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 4.4 MB/s | 5.4 MB 00:01 2023-07-20T18:56:30.760 INFO:teuthology.orchestra.run.smithi156.stdout:(5/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 4.2 MB/s | 2.3 MB 00:00 2023-07-20T18:56:30.944 INFO:teuthology.orchestra.run.smithi156.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 6.7 MB/s | 1.6 MB 00:00 2023-07-20T18:56:31.444 INFO:teuthology.orchestra.run.smithi156.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 6.7 MB/s | 4.6 MB 00:00 2023-07-20T18:56:31.666 INFO:teuthology.orchestra.run.smithi061.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:56:31.667 INFO:teuthology.orchestra.run.smithi061.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:56:31.682 INFO:teuthology.orchestra.run.smithi061.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T18:56:31.683 INFO:teuthology.orchestra.run.smithi061.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T18:56:31.684 INFO:teuthology.orchestra.run.smithi061.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T18:56:31.686 INFO:teuthology.orchestra.run.smithi061.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T18:56:31.688 INFO:teuthology.orchestra.run.smithi061.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T18:56:31.691 INFO:teuthology.orchestra.run.smithi061.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T18:56:31.692 INFO:teuthology.orchestra.run.smithi061.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T18:56:31.834 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-07-20T18:56:31.848 INFO:teuthology.orchestra.run.smithi061.stdout:=========================================================================================================================== 2023-07-20T18:56:31.848 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-07-20T18:56:31.848 INFO:teuthology.orchestra.run.smithi061.stdout:=========================================================================================================================== 2023-07-20T18:56:31.848 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-07-20T18:56:31.848 INFO:teuthology.orchestra.run.smithi061.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T18:56:31.849 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T18:56:31.850 INFO:teuthology.orchestra.run.smithi061.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:56:31.851 INFO:teuthology.orchestra.run.smithi061.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T18:56:31.852 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrading: 2023-07-20T18:56:31.853 INFO:teuthology.orchestra.run.smithi061.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T18:56:31.854 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T18:56:31.855 INFO:teuthology.orchestra.run.smithi061.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T18:56:31.856 INFO:teuthology.orchestra.run.smithi061.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T18:56:31.857 INFO:teuthology.orchestra.run.smithi061.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T18:56:31.858 INFO:teuthology.orchestra.run.smithi061.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T18:56:31.859 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T18:56:31.860 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T18:56:31.861 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T18:56:31.862 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T18:56:31.863 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T18:56:31.864 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T18:56:31.865 INFO:teuthology.orchestra.run.smithi061.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T18:56:31.865 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T18:56:31.865 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T18:56:31.865 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T18:56:31.865 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T18:56:31.866 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling module streams: 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql 10 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout:=========================================================================================================================== 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout:Install 128 Packages 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrade 3 Packages 2023-07-20T18:56:31.867 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:56:31.868 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 206 M 2023-07-20T18:56:31.868 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-07-20T18:56:32.346 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:56:32.347 INFO:teuthology.orchestra.run.smithi149.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T18:56:32.361 INFO:teuthology.orchestra.run.smithi149.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T18:56:32.362 INFO:teuthology.orchestra.run.smithi149.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T18:56:32.363 INFO:teuthology.orchestra.run.smithi149.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T18:56:32.364 INFO:teuthology.orchestra.run.smithi149.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T18:56:32.366 INFO:teuthology.orchestra.run.smithi149.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T18:56:32.369 INFO:teuthology.orchestra.run.smithi149.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T18:56:32.370 INFO:teuthology.orchestra.run.smithi149.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T18:56:32.420 INFO:teuthology.orchestra.run.smithi156.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 7.6 MB/s | 22 MB 00:02 2023-07-20T18:56:32.460 INFO:teuthology.orchestra.run.smithi061.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 32 kB/s | 6.4 kB 00:00 2023-07-20T18:56:32.502 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-07-20T18:56:32.515 INFO:teuthology.orchestra.run.smithi149.stdout:=========================================================================================================================== 2023-07-20T18:56:32.515 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-07-20T18:56:32.515 INFO:teuthology.orchestra.run.smithi149.stdout:=========================================================================================================================== 2023-07-20T18:56:32.515 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-07-20T18:56:32.516 INFO:teuthology.orchestra.run.smithi149.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T18:56:32.516 INFO:teuthology.orchestra.run.smithi149.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T18:56:32.516 INFO:teuthology.orchestra.run.smithi149.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T18:56:32.516 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T18:56:32.516 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T18:56:32.516 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T18:56:32.517 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T18:56:32.518 INFO:teuthology.orchestra.run.smithi149.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T18:56:32.519 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T18:56:32.520 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrading: 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T18:56:32.521 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T18:56:32.522 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T18:56:32.523 INFO:teuthology.orchestra.run.smithi149.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T18:56:32.524 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T18:56:32.525 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T18:56:32.526 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T18:56:32.527 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T18:56:32.528 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T18:56:32.529 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T18:56:32.529 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T18:56:32.529 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T18:56:32.529 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T18:56:32.530 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T18:56:32.530 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:56:32.530 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T18:56:32.530 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T18:56:32.530 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T18:56:32.530 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T18:56:32.531 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T18:56:32.532 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T18:56:32.533 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T18:56:32.534 INFO:teuthology.orchestra.run.smithi149.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout:Enabling module streams: 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql 10 2023-07-20T18:56:32.535 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:56:32.536 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-07-20T18:56:32.536 INFO:teuthology.orchestra.run.smithi149.stdout:=========================================================================================================================== 2023-07-20T18:56:32.536 INFO:teuthology.orchestra.run.smithi149.stdout:Install 128 Packages 2023-07-20T18:56:32.536 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrade 3 Packages 2023-07-20T18:56:32.536 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:56:32.537 INFO:teuthology.orchestra.run.smithi156.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 256 kB/s | 24 kB 00:00 2023-07-20T18:56:32.537 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 206 M 2023-07-20T18:56:32.537 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-07-20T18:56:32.728 INFO:teuthology.orchestra.run.smithi156.stdout:(10/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 10 MB/s | 18 MB 00:01 2023-07-20T18:56:32.829 INFO:teuthology.orchestra.run.smithi156.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 2.5 MB/s | 260 kB 00:00 2023-07-20T18:56:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-07-20T18:56:32.996 INFO:teuthology.orchestra.run.smithi156.stdout:(12/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 8.7 MB/s | 14 MB 00:01 2023-07-20T18:56:33.012 INFO:teuthology.orchestra.run.smithi156.stdout:(13/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 169 kB/s | 31 kB 00:00 2023-07-20T18:56:33.105 INFO:teuthology.orchestra.run.smithi156.stdout:(14/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 7.0 MB/s | 771 kB 00:00 2023-07-20T18:56:33.130 INFO:teuthology.orchestra.run.smithi156.stdout:(15/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.2 MB/s | 139 kB 00:00 2023-07-20T18:56:33.188 INFO:teuthology.orchestra.run.smithi156.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.6 MB/s | 139 kB 00:00 2023-07-20T18:56:33.222 INFO:teuthology.orchestra.run.smithi156.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 5.0 MB/s | 472 kB 00:00 2023-07-20T18:56:33.330 INFO:teuthology.orchestra.run.smithi156.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 413 kB/s | 45 kB 00:00 2023-07-20T18:56:33.335 INFO:teuthology.orchestra.run.smithi061.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 971 kB/s | 850 kB 00:00 2023-07-20T18:56:33.464 INFO:teuthology.orchestra.run.smithi156.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 919 kB/s | 122 kB 00:00 2023-07-20T18:56:33.535 INFO:teuthology.orchestra.run.smithi061.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 835 kB/s | 167 kB 00:00 2023-07-20T18:56:33.557 INFO:teuthology.orchestra.run.smithi149.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 850 kB 00:00 2023-07-20T18:56:33.573 INFO:teuthology.orchestra.run.smithi156.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.8 MB/s | 197 kB 00:00 2023-07-20T18:56:33.698 INFO:teuthology.orchestra.run.smithi156.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.9 MB/s | 370 kB 00:00 2023-07-20T18:56:33.757 INFO:teuthology.orchestra.run.smithi149.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 836 kB/s | 167 kB 00:00 2023-07-20T18:56:33.823 INFO:teuthology.orchestra.run.smithi156.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.8 MB/s | 363 kB 00:00 2023-07-20T18:56:33.974 INFO:teuthology.orchestra.run.smithi156.stdout:(23/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 8.6 MB/s | 6.8 MB 00:00 2023-07-20T18:56:33.999 INFO:teuthology.orchestra.run.smithi156.stdout:(24/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 608 kB/s | 107 kB 00:00 2023-07-20T18:56:34.049 INFO:teuthology.orchestra.run.smithi156.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 94 kB 00:00 2023-07-20T18:56:34.408 INFO:teuthology.orchestra.run.smithi156.stdout:(26/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 8.6 MB/s | 3.5 MB 00:00 2023-07-20T18:56:34.475 INFO:teuthology.orchestra.run.smithi156.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 346 kB/s | 23 kB 00:00 2023-07-20T18:56:34.551 INFO:teuthology.orchestra.run.smithi156.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.7 MB/s | 129 kB 00:00 2023-07-20T18:56:34.691 INFO:teuthology.orchestra.run.smithi149.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 2.5 MB/s | 2.3 MB 00:00 2023-07-20T18:56:34.743 INFO:teuthology.orchestra.run.smithi156.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 9.0 MB/s | 1.7 MB 00:00 2023-07-20T18:56:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 2.3 MB 00:01 2023-07-20T18:56:34.768 INFO:teuthology.orchestra.run.smithi156.stdout:(30/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 249 kB/s | 179 kB 00:00 2023-07-20T18:56:34.893 INFO:teuthology.orchestra.run.smithi156.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.9 MB/s | 241 kB 00:00 2023-07-20T18:56:35.018 INFO:teuthology.orchestra.run.smithi156.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 380 kB/s | 47 kB 00:00 2023-07-20T18:56:35.127 INFO:teuthology.orchestra.run.smithi061.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.9 MB/s | 5.4 MB 00:02 2023-07-20T18:56:35.135 INFO:teuthology.orchestra.run.smithi156.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 125 kB/s | 15 kB 00:00 2023-07-20T18:56:35.294 INFO:teuthology.orchestra.run.smithi156.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.6 MB/s | 258 kB 00:00 2023-07-20T18:56:35.428 INFO:teuthology.orchestra.run.smithi156.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.2 MB/s | 165 kB 00:00 2023-07-20T18:56:35.461 INFO:teuthology.orchestra.run.smithi061.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 2.2 MB/s | 1.6 MB 00:00 2023-07-20T18:56:35.533 INFO:teuthology.orchestra.run.smithi149.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 1.8 MB/s | 1.6 MB 00:00 2023-07-20T18:56:35.561 INFO:teuthology.orchestra.run.smithi156.stdout:(36/131): ceph-mgr-diskprediction-local-18.0.0- 9.1 MB/s | 7.4 MB 00:00 2023-07-20T18:56:35.587 INFO:teuthology.orchestra.run.smithi156.stdout:(37/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 113 kB/s | 18 kB 00:00 2023-07-20T18:56:35.987 INFO:teuthology.orchestra.run.smithi156.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 147 kB/s | 59 kB 00:00 2023-07-20T18:56:36.096 INFO:teuthology.orchestra.run.smithi156.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 693 kB/s | 75 kB 00:00 2023-07-20T18:56:36.261 INFO:teuthology.orchestra.run.smithi061.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 4.1 MB/s | 4.6 MB 00:01 2023-07-20T18:56:36.446 INFO:teuthology.orchestra.run.smithi156.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.3 MB/s | 476 kB 00:00 2023-07-20T18:56:36.513 INFO:teuthology.orchestra.run.smithi156.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 656 kB/s | 44 kB 00:00 2023-07-20T18:56:36.524 INFO:teuthology.orchestra.run.smithi149.stdout:(6/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.4 MB/s | 5.4 MB 00:03 2023-07-20T18:56:36.663 INFO:teuthology.orchestra.run.smithi156.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-07-20T18:56:36.855 INFO:teuthology.orchestra.run.smithi156.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 172 kB/s | 33 kB 00:00 2023-07-20T18:56:36.939 INFO:teuthology.orchestra.run.smithi156.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 196 kB/s | 16 kB 00:00 2023-07-20T18:56:37.139 INFO:teuthology.orchestra.run.smithi156.stdout:(45/131): gperftools-libs-2.7-9.el8.x86_64.rpm 194 kB/s | 306 kB 00:01 2023-07-20T18:56:37.456 INFO:teuthology.orchestra.run.smithi156.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 334 kB/s | 173 kB 00:00 2023-07-20T18:56:37.603 INFO:teuthology.orchestra.run.smithi156.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 556 kB/s | 81 kB 00:00 2023-07-20T18:56:37.673 INFO:teuthology.orchestra.run.smithi156.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 158 kB/s | 11 kB 00:00 2023-07-20T18:56:37.740 INFO:teuthology.orchestra.run.smithi156.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 265 kB/s | 18 kB 00:00 2023-07-20T18:56:38.108 INFO:teuthology.orchestra.run.smithi149.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 1.8 MB/s | 4.6 MB 00:02 2023-07-20T18:56:38.282 INFO:teuthology.orchestra.run.smithi156.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 336 kB/s | 384 kB 00:01 2023-07-20T18:56:38.475 INFO:teuthology.orchestra.run.smithi156.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 254 kB/s | 49 kB 00:00 2023-07-20T18:56:39.041 INFO:teuthology.orchestra.run.smithi156.stdout:(52/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 7.4 MB/s | 48 MB 00:06 2023-07-20T18:56:39.067 INFO:teuthology.orchestra.run.smithi156.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 99 kB/s | 59 kB 00:00 2023-07-20T18:56:39.100 INFO:teuthology.orchestra.run.smithi156.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 878 kB/s | 1.2 MB 00:01 2023-07-20T18:56:39.175 INFO:teuthology.orchestra.run.smithi156.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 221 kB/s | 16 kB 00:00 2023-07-20T18:56:39.414 INFO:teuthology.orchestra.run.smithi156.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 141 kB/s | 34 kB 00:00 2023-07-20T18:56:39.443 INFO:teuthology.orchestra.run.smithi156.stdout:(57/131): python3-natsort-7.1.1-2.el8.noarch.rp 150 kB/s | 60 kB 00:00 2023-07-20T18:56:39.444 INFO:teuthology.orchestra.run.smithi061.stdout:(8/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el8 4.3 MB/s | 14 MB 00:03 2023-07-20T18:56:39.520 INFO:teuthology.orchestra.run.smithi061.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 321 kB/s | 24 kB 00:00 2023-07-20T18:56:39.577 INFO:teuthology.orchestra.run.smithi156.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 461 kB/s | 61 kB 00:00 2023-07-20T18:56:39.602 INFO:teuthology.orchestra.run.smithi156.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.0 MB/s | 196 kB 00:00 2023-07-20T18:56:39.644 INFO:teuthology.orchestra.run.smithi156.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 281 kB/s | 19 kB 00:00 2023-07-20T18:56:39.669 INFO:teuthology.orchestra.run.smithi156.stdout:(61/131): python3-pecan-1.3.2-9.el8.noarch.rpm 470 kB/s | 283 kB 00:00 2023-07-20T18:56:39.694 INFO:teuthology.orchestra.run.smithi156.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2023-07-20T18:56:39.720 INFO:teuthology.orchestra.run.smithi156.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 387 kB/s | 29 kB 00:00 2023-07-20T18:56:39.745 INFO:teuthology.orchestra.run.smithi156.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2023-07-20T18:56:39.870 INFO:teuthology.orchestra.run.smithi156.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 488 kB/s | 61 kB 00:00 2023-07-20T18:56:39.962 INFO:teuthology.orchestra.run.smithi156.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 954 kB/s | 255 kB 00:00 2023-07-20T18:56:40.046 INFO:teuthology.orchestra.run.smithi156.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 275 kB/s | 23 kB 00:00 2023-07-20T18:56:40.088 INFO:teuthology.orchestra.run.smithi156.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 393 kB/s | 85 kB 00:00 2023-07-20T18:56:40.105 INFO:teuthology.orchestra.run.smithi156.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-07-20T18:56:40.213 INFO:teuthology.orchestra.run.smithi156.stdout:(70/131): python3-webob-1.8.5-1.el8.1.noarch.rp 509 kB/s | 251 kB 00:00 2023-07-20T18:56:40.322 INFO:teuthology.orchestra.run.smithi156.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 216 kB/s | 47 kB 00:00 2023-07-20T18:56:40.423 INFO:teuthology.orchestra.run.smithi156.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 958 kB/s | 200 kB 00:00 2023-07-20T18:56:40.506 INFO:teuthology.orchestra.run.smithi156.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2023-07-20T18:56:40.565 INFO:teuthology.orchestra.run.smithi156.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2023-07-20T18:56:40.587 INFO:teuthology.orchestra.run.smithi061.stdout:(10/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 3.5 MB/s | 18 MB 00:05 2023-07-20T18:56:40.598 INFO:teuthology.orchestra.run.smithi156.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 524 kB/s | 48 kB 00:00 2023-07-20T18:56:40.665 INFO:teuthology.orchestra.run.smithi156.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 867 kB/s | 87 kB 00:00 2023-07-20T18:56:40.716 INFO:teuthology.orchestra.run.smithi156.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 365 kB/s | 43 kB 00:00 2023-07-20T18:56:40.737 INFO:teuthology.orchestra.run.smithi061.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.7 MB/s | 260 kB 00:00 2023-07-20T18:56:40.750 INFO:teuthology.orchestra.run.smithi156.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 1.3 MB/s | 109 kB 00:00 2023-07-20T18:56:40.775 INFO:teuthology.orchestra.run.smithi156.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2023-07-20T18:56:40.812 INFO:teuthology.orchestra.run.smithi061.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 414 kB/s | 31 kB 00:00 2023-07-20T18:56:40.875 INFO:teuthology.orchestra.run.smithi156.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.4 MB/s | 560 kB 00:00 2023-07-20T18:56:40.925 INFO:teuthology.orchestra.run.smithi156.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 124 kB/s | 19 kB 00:00 2023-07-20T18:56:41.001 INFO:teuthology.orchestra.run.smithi156.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.2 MB/s | 155 kB 00:00 2023-07-20T18:56:41.087 INFO:teuthology.orchestra.run.smithi061.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 2.7 MB/s | 771 kB 00:00 2023-07-20T18:56:41.093 INFO:teuthology.orchestra.run.smithi156.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 3.3 MB/s | 559 kB 00:00 2023-07-20T18:56:41.110 INFO:teuthology.orchestra.run.smithi156.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 165 kB/s | 18 kB 00:00 2023-07-20T18:56:41.193 INFO:teuthology.orchestra.run.smithi156.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-07-20T18:56:41.204 INFO:teuthology.orchestra.run.smithi061.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.2 MB/s | 139 kB 00:00 2023-07-20T18:56:41.235 INFO:teuthology.orchestra.run.smithi156.stdout:(86/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.2 MB/s | 605 kB 00:00 2023-07-20T18:56:41.288 INFO:teuthology.orchestra.run.smithi061.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.6 MB/s | 139 kB 00:00 2023-07-20T18:56:41.377 INFO:teuthology.orchestra.run.smithi156.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 2.3 MB 00:00 2023-07-20T18:56:41.419 INFO:teuthology.orchestra.run.smithi156.stdout:(88/131): thrift-0.13.0-2.el8.x86_64.rpm 1.3 MB/s | 1.7 MB 00:01 2023-07-20T18:56:41.445 INFO:teuthology.orchestra.run.smithi156.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 346 kB/s | 72 kB 00:00 2023-07-20T18:56:41.496 INFO:teuthology.orchestra.run.smithi061.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.2 MB/s | 472 kB 00:00 2023-07-20T18:56:41.520 INFO:teuthology.orchestra.run.smithi156.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 583 kB/s | 83 kB 00:00 2023-07-20T18:56:41.545 INFO:teuthology.orchestra.run.smithi156.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 476 kB/s | 60 kB 00:00 2023-07-20T18:56:41.596 INFO:teuthology.orchestra.run.smithi156.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 678 kB/s | 98 kB 00:00 2023-07-20T18:56:41.621 INFO:teuthology.orchestra.run.smithi156.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 514 kB/s | 39 kB 00:00 2023-07-20T18:56:41.663 INFO:teuthology.orchestra.run.smithi156.stdout:(94/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 404 kB/s | 58 kB 00:00 2023-07-20T18:56:41.713 INFO:teuthology.orchestra.run.smithi156.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.4 MB/s | 644 kB 00:00 2023-07-20T18:56:41.772 INFO:teuthology.orchestra.run.smithi156.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-07-20T18:56:41.805 INFO:teuthology.orchestra.run.smithi156.stdout:(97/131): NetworkManager-initscripts-updown-1.3 967 kB/s | 138 kB 00:00 2023-07-20T18:56:41.922 INFO:teuthology.orchestra.run.smithi156.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 465 kB/s | 54 kB 00:00 2023-07-20T18:56:41.948 INFO:teuthology.orchestra.run.smithi156.stdout:(99/131): python3-pyasn1-modules-0.3.7-6.el8.no 627 kB/s | 110 kB 00:00 2023-07-20T18:56:42.131 INFO:teuthology.orchestra.run.smithi156.stdout:(100/131): python3-scipy-1.0.0-21.module+el8.5. 33 MB/s | 14 MB 00:00 2023-07-20T18:56:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:(101/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2023-07-20T18:56:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:(102/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2023-07-20T18:56:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:(103/131): python3-markupsafe-0.23-19.el8.x86_6 310 kB/s | 39 kB 00:00 2023-07-20T18:56:42.299 INFO:teuthology.orchestra.run.smithi156.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 4.5 MB/s | 538 kB 00:00 2023-07-20T18:56:42.408 INFO:teuthology.orchestra.run.smithi156.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-07-20T18:56:42.441 INFO:teuthology.orchestra.run.smithi156.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 852 kB/s | 157 kB 00:00 2023-07-20T18:56:42.542 INFO:teuthology.orchestra.run.smithi156.stdout:(107/131): bison-3.0.4-10.el8.x86_64.rpm 1.7 MB/s | 688 kB 00:00 2023-07-20T18:56:42.575 INFO:teuthology.orchestra.run.smithi156.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-07-20T18:56:42.698 INFO:teuthology.orchestra.run.smithi156.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-07-20T18:56:42.726 INFO:teuthology.orchestra.run.smithi156.stdout:(110/131): python3-jsonpatch-1.21-2.el8.noarch. 144 kB/s | 27 kB 00:00 2023-07-20T18:56:42.751 INFO:teuthology.orchestra.run.smithi156.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.8 MB/s | 320 kB 00:00 2023-07-20T18:56:42.843 INFO:teuthology.orchestra.run.smithi156.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 304 kB/s | 44 kB 00:00 2023-07-20T18:56:42.918 INFO:teuthology.orchestra.run.smithi156.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 19 MB/s | 3.7 MB 00:00 2023-07-20T18:56:42.952 INFO:teuthology.orchestra.run.smithi156.stdout:(114/131): python3-werkzeug-0.12.2-4.el8.noarch 2.2 MB/s | 457 kB 00:00 2023-07-20T18:56:42.977 INFO:teuthology.orchestra.run.smithi156.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2023-07-20T18:56:43.002 INFO:teuthology.orchestra.run.smithi156.stdout:(116/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.4 MB/s | 225 kB 00:00 2023-07-20T18:56:43.078 INFO:teuthology.orchestra.run.smithi156.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.0 MB/s | 126 kB 00:00 2023-07-20T18:56:43.103 INFO:teuthology.orchestra.run.smithi156.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 159 kB/s | 20 kB 00:00 2023-07-20T18:56:43.159 INFO:teuthology.orchestra.run.smithi156.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 660 kB/s | 103 kB 00:00 2023-07-20T18:56:43.270 INFO:teuthology.orchestra.run.smithi156.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 3.6 MB/s | 709 kB 00:00 2023-07-20T18:56:43.354 INFO:teuthology.orchestra.run.smithi156.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 19 MB/s | 4.9 MB 00:00 2023-07-20T18:56:43.379 INFO:teuthology.orchestra.run.smithi156.stdout:(122/131): libpq-13.5-1.el8.x86_64.rpm 901 kB/s | 198 kB 00:00 2023-07-20T18:56:43.446 INFO:teuthology.orchestra.run.smithi156.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 2.1 MB/s | 202 kB 00:00 2023-07-20T18:56:43.505 INFO:teuthology.orchestra.run.smithi061.stdout:(17/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 3.4 MB/s | 6.8 MB 00:02 2023-07-20T18:56:43.513 INFO:teuthology.orchestra.run.smithi156.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-07-20T18:56:43.538 INFO:teuthology.orchestra.run.smithi156.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-07-20T18:56:43.572 INFO:teuthology.orchestra.run.smithi061.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 673 kB/s | 45 kB 00:00 2023-07-20T18:56:43.638 INFO:teuthology.orchestra.run.smithi156.stdout:(126/131): postgresql-contrib-10.21-2.module+el 6.3 MB/s | 808 kB 00:00 2023-07-20T18:56:43.667 INFO:teuthology.orchestra.run.smithi061.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.3 MB/s | 122 kB 00:00 2023-07-20T18:56:43.739 INFO:teuthology.orchestra.run.smithi156.stdout:(127/131): postgresql-server-10.21-2.module+el8 17 MB/s | 5.1 MB 00:00 2023-07-20T18:56:43.781 INFO:teuthology.orchestra.run.smithi156.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.2 MB/s | 1.5 MB 00:00 2023-07-20T18:56:43.891 INFO:teuthology.orchestra.run.smithi149.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 2.0 MB/s | 22 MB 00:11 2023-07-20T18:56:43.939 INFO:teuthology.orchestra.run.smithi156.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.9 MB/s | 798 kB 00:00 2023-07-20T18:56:43.965 INFO:teuthology.orchestra.run.smithi061.stdout:(20/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 1.9 MB/s | 22 MB 00:11 2023-07-20T18:56:43.966 INFO:teuthology.orchestra.run.smithi149.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 325 kB/s | 24 kB 00:00 2023-07-20T18:56:43.990 INFO:teuthology.orchestra.run.smithi061.stdout:(21/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 611 kB/s | 197 kB 00:00 2023-07-20T18:56:44.117 INFO:teuthology.orchestra.run.smithi149.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 2.3 MB/s | 14 MB 00:06 2023-07-20T18:56:44.249 INFO:teuthology.orchestra.run.smithi061.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 363 kB 00:00 2023-07-20T18:56:44.265 INFO:teuthology.orchestra.run.smithi156.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 6.9 MB/s | 3.6 MB 00:00 2023-07-20T18:56:44.273 INFO:teuthology.orchestra.run.smithi061.stdout:(23/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.2 MB/s | 370 kB 00:00 2023-07-20T18:56:44.284 INFO:teuthology.orchestra.run.smithi149.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.5 MB/s | 260 kB 00:00 2023-07-20T18:56:44.324 INFO:teuthology.orchestra.run.smithi061.stdout:(24/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 107 kB 00:00 2023-07-20T18:56:44.351 INFO:teuthology.orchestra.run.smithi149.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 466 kB/s | 31 kB 00:00 2023-07-20T18:56:44.374 INFO:teuthology.orchestra.run.smithi061.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 936 kB/s | 94 kB 00:00 2023-07-20T18:56:44.415 INFO:teuthology.orchestra.run.smithi156.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 4.7 MB/s | 3.7 MB 00:00 2023-07-20T18:56:44.419 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:56:44.419 INFO:teuthology.orchestra.run.smithi156.stdout:Total 14 MB/s | 206 MB 00:15 2023-07-20T18:56:44.483 INFO:teuthology.orchestra.run.smithi061.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.6 MB/s | 179 kB 00:00 2023-07-20T18:56:44.541 INFO:teuthology.orchestra.run.smithi061.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 396 kB/s | 23 kB 00:00 2023-07-20T18:56:44.608 INFO:teuthology.orchestra.run.smithi061.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.9 MB/s | 129 kB 00:00 2023-07-20T18:56:44.618 INFO:teuthology.orchestra.run.smithi149.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 2.8 MB/s | 771 kB 00:00 2023-07-20T18:56:44.743 INFO:teuthology.orchestra.run.smithi149.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.1 MB/s | 139 kB 00:00 2023-07-20T18:56:44.860 INFO:teuthology.orchestra.run.smithi149.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.2 MB/s | 139 kB 00:00 2023-07-20T18:56:45.102 INFO:teuthology.orchestra.run.smithi149.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 1.9 MB/s | 472 kB 00:00 2023-07-20T18:56:45.150 INFO:teuthology.orchestra.run.smithi061.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 3.2 MB/s | 1.7 MB 00:00 2023-07-20T18:56:45.264 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-07-20T18:56:45.376 INFO:teuthology.orchestra.run.smithi061.stdout:(30/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 3.4 MB/s | 3.5 MB 00:01 2023-07-20T18:56:45.384 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-07-20T18:56:45.384 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-07-20T18:56:45.518 INFO:teuthology.orchestra.run.smithi061.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.7 MB/s | 241 kB 00:00 2023-07-20T18:56:45.593 INFO:teuthology.orchestra.run.smithi061.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 632 kB/s | 47 kB 00:00 2023-07-20T18:56:45.602 INFO:teuthology.orchestra.run.smithi149.stdout:(17/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 2.0 MB/s | 18 MB 00:09 2023-07-20T18:56:45.669 INFO:teuthology.orchestra.run.smithi149.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 673 kB/s | 45 kB 00:00 2023-07-20T18:56:45.676 INFO:teuthology.orchestra.run.smithi061.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 175 kB/s | 15 kB 00:00 2023-07-20T18:56:45.736 INFO:teuthology.orchestra.run.smithi149.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.8 MB/s | 122 kB 00:00 2023-07-20T18:56:45.793 INFO:teuthology.orchestra.run.smithi061.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.2 MB/s | 258 kB 00:00 2023-07-20T18:56:45.845 INFO:teuthology.orchestra.run.smithi149.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.8 MB/s | 197 kB 00:00 2023-07-20T18:56:45.869 INFO:teuthology.orchestra.run.smithi061.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 2.2 MB/s | 165 kB 00:00 2023-07-20T18:56:45.936 INFO:teuthology.orchestra.run.smithi061.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 270 kB/s | 18 kB 00:00 2023-07-20T18:56:46.003 INFO:teuthology.orchestra.run.smithi149.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.3 MB/s | 370 kB 00:00 2023-07-20T18:56:46.162 INFO:teuthology.orchestra.run.smithi149.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.2 MB/s | 363 kB 00:00 2023-07-20T18:56:46.229 INFO:teuthology.orchestra.run.smithi149.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.6 MB/s | 107 kB 00:00 2023-07-20T18:56:46.296 INFO:teuthology.orchestra.run.smithi149.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.4 MB/s | 94 kB 00:00 2023-07-20T18:56:46.403 INFO:teuthology.orchestra.run.smithi061.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 656 kB/s | 306 kB 00:00 2023-07-20T18:56:46.512 INFO:teuthology.orchestra.run.smithi061.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 543 kB/s | 59 kB 00:00 2023-07-20T18:56:46.512 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-07-20T18:56:46.516 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-07-20T18:56:46.620 INFO:teuthology.orchestra.run.smithi061.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 694 kB/s | 75 kB 00:00 2023-07-20T18:56:46.812 INFO:teuthology.orchestra.run.smithi061.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.4 MB/s | 476 kB 00:00 2023-07-20T18:56:46.912 INFO:teuthology.orchestra.run.smithi061.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 436 kB/s | 44 kB 00:00 2023-07-20T18:56:47.013 INFO:teuthology.orchestra.run.smithi061.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-07-20T18:56:47.113 INFO:teuthology.orchestra.run.smithi061.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 331 kB/s | 33 kB 00:00 2023-07-20T18:56:47.247 INFO:teuthology.orchestra.run.smithi061.stdout:(44/131): ceph-mgr-diskprediction-local-18.0.0- 3.5 MB/s | 7.4 MB 00:02 2023-07-20T18:56:47.272 INFO:teuthology.orchestra.run.smithi061.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 103 kB/s | 16 kB 00:00 2023-07-20T18:56:47.431 INFO:teuthology.orchestra.run.smithi149.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 3.1 MB/s | 3.5 MB 00:01 2023-07-20T18:56:47.447 INFO:teuthology.orchestra.run.smithi061.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.1 MB/s | 384 kB 00:00 2023-07-20T18:56:47.513 INFO:teuthology.orchestra.run.smithi149.stdout:(26/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 2.8 MB/s | 6.8 MB 00:02 2023-07-20T18:56:47.548 INFO:teuthology.orchestra.run.smithi061.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 813 kB/s | 81 kB 00:00 2023-07-20T18:56:47.563 INFO:teuthology.orchestra.run.smithi149.stdout:(27/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 1.3 MB/s | 179 kB 00:00 2023-07-20T18:56:47.589 INFO:teuthology.orchestra.run.smithi149.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 307 kB/s | 23 kB 00:00 2023-07-20T18:56:47.622 INFO:teuthology.orchestra.run.smithi149.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 2.2 MB/s | 129 kB 00:00 2023-07-20T18:56:47.631 INFO:teuthology.orchestra.run.smithi061.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 449 kB/s | 173 kB 00:00 2023-07-20T18:56:47.656 INFO:teuthology.orchestra.run.smithi061.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-07-20T18:56:47.732 INFO:teuthology.orchestra.run.smithi061.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-07-20T18:56:47.832 INFO:teuthology.orchestra.run.smithi061.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 488 kB/s | 49 kB 00:00 2023-07-20T18:56:47.916 INFO:teuthology.orchestra.run.smithi061.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 4.5 MB/s | 1.2 MB 00:00 2023-07-20T18:56:47.941 INFO:teuthology.orchestra.run.smithi061.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 541 kB/s | 59 kB 00:00 2023-07-20T18:56:48.025 INFO:teuthology.orchestra.run.smithi061.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 555 kB/s | 60 kB 00:00 2023-07-20T18:56:48.050 INFO:teuthology.orchestra.run.smithi061.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-07-20T18:56:48.114 INFO:teuthology.orchestra.run.smithi149.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 3.3 MB/s | 1.7 MB 00:00 2023-07-20T18:56:48.134 INFO:teuthology.orchestra.run.smithi061.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-07-20T18:56:48.151 INFO:teuthology.orchestra.run.smithi061.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 336 kB/s | 34 kB 00:00 2023-07-20T18:56:48.243 INFO:teuthology.orchestra.run.smithi061.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2023-07-20T18:56:48.256 INFO:teuthology.orchestra.run.smithi149.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.7 MB/s | 241 kB 00:00 2023-07-20T18:56:48.268 INFO:teuthology.orchestra.run.smithi061.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 527 kB/s | 61 kB 00:00 2023-07-20T18:56:48.343 INFO:teuthology.orchestra.run.smithi061.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 187 kB/s | 19 kB 00:00 2023-07-20T18:56:48.356 INFO:teuthology.orchestra.run.smithi149.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 474 kB/s | 47 kB 00:00 2023-07-20T18:56:48.377 INFO:teuthology.orchestra.run.smithi061.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 220 kB/s | 24 kB 00:00 2023-07-20T18:56:48.432 INFO:teuthology.orchestra.run.smithi149.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 194 kB/s | 15 kB 00:00 2023-07-20T18:56:48.444 INFO:teuthology.orchestra.run.smithi061.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 292 kB/s | 29 kB 00:00 2023-07-20T18:56:48.486 INFO:teuthology.orchestra.run.smithi061.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-07-20T18:56:48.540 INFO:teuthology.orchestra.run.smithi149.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.3 MB/s | 258 kB 00:00 2023-07-20T18:56:48.544 INFO:teuthology.orchestra.run.smithi061.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.5 MB/s | 255 kB 00:00 2023-07-20T18:56:48.607 INFO:teuthology.orchestra.run.smithi149.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 2.4 MB/s | 165 kB 00:00 2023-07-20T18:56:48.612 INFO:teuthology.orchestra.run.smithi061.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-07-20T18:56:48.628 INFO:teuthology.orchestra.run.smithi061.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 732 kB/s | 61 kB 00:00 2023-07-20T18:56:48.666 INFO:teuthology.orchestra.run.smithi149.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 308 kB/s | 18 kB 00:00 2023-07-20T18:56:48.712 INFO:teuthology.orchestra.run.smithi061.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 853 kB/s | 85 kB 00:00 2023-07-20T18:56:48.729 INFO:teuthology.orchestra.run.smithi061.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 229 kB/s | 23 kB 00:00 2023-07-20T18:56:48.754 INFO:teuthology.orchestra.run.smithi061.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T18:56:48.833 INFO:teuthology.orchestra.run.smithi149.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-07-20T18:56:48.875 INFO:teuthology.orchestra.run.smithi149.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-07-20T18:56:48.917 INFO:teuthology.orchestra.run.smithi149.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-07-20T18:56:48.979 INFO:teuthology.orchestra.run.smithi061.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 208 kB/s | 47 kB 00:00 2023-07-20T18:56:49.050 INFO:teuthology.orchestra.run.smithi149.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.5 MB/s | 476 kB 00:00 2023-07-20T18:56:49.101 INFO:teuthology.orchestra.run.smithi149.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 876 kB/s | 44 kB 00:00 2023-07-20T18:56:49.113 INFO:teuthology.orchestra.run.smithi061.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 4.3 MB/s | 1.7 MB 00:00 2023-07-20T18:56:49.163 INFO:teuthology.orchestra.run.smithi061.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.1 MB/s | 200 kB 00:00 2023-07-20T18:56:49.168 INFO:teuthology.orchestra.run.smithi149.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-07-20T18:56:49.201 INFO:teuthology.orchestra.run.smithi149.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 994 kB/s | 33 kB 00:00 2023-07-20T18:56:49.235 INFO:teuthology.orchestra.run.smithi149.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 489 kB/s | 16 kB 00:00 2023-07-20T18:56:49.314 INFO:teuthology.orchestra.run.smithi061.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 213 kB/s | 43 kB 00:00 2023-07-20T18:56:49.339 INFO:teuthology.orchestra.run.smithi061.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2023-07-20T18:56:49.398 INFO:teuthology.orchestra.run.smithi061.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2023-07-20T18:56:49.448 INFO:teuthology.orchestra.run.smithi061.stdout:(76/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 359 kB/s | 48 kB 00:00 2023-07-20T18:56:49.486 INFO:teuthology.orchestra.run.smithi149.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 690 kB/s | 173 kB 00:00 2023-07-20T18:56:49.490 INFO:teuthology.orchestra.run.smithi061.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 464 kB/s | 43 kB 00:00 2023-07-20T18:56:49.557 INFO:teuthology.orchestra.run.smithi061.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 1.0 MB/s | 109 kB 00:00 2023-07-20T18:56:49.586 INFO:teuthology.orchestra.run.smithi149.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.8 MB/s | 384 kB 00:00 2023-07-20T18:56:49.591 INFO:teuthology.orchestra.run.smithi061.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2023-07-20T18:56:49.628 INFO:teuthology.orchestra.run.smithi149.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.9 MB/s | 81 kB 00:00 2023-07-20T18:56:49.661 INFO:teuthology.orchestra.run.smithi149.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 334 kB/s | 11 kB 00:00 2023-07-20T18:56:49.699 INFO:teuthology.orchestra.run.smithi061.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.9 MB/s | 560 kB 00:00 2023-07-20T18:56:49.725 INFO:teuthology.orchestra.run.smithi061.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 140 kB/s | 19 kB 00:00 2023-07-20T18:56:49.770 INFO:teuthology.orchestra.run.smithi149.stdout:(49/131): ceph-mgr-diskprediction-local-18.0.0- 3.5 MB/s | 7.4 MB 00:02 2023-07-20T18:56:49.795 INFO:teuthology.orchestra.run.smithi149.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 131 kB/s | 18 kB 00:00 2023-07-20T18:56:49.808 INFO:teuthology.orchestra.run.smithi061.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 6.6 MB/s | 559 kB 00:00 2023-07-20T18:56:49.834 INFO:teuthology.orchestra.run.smithi061.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2023-07-20T18:56:49.838 INFO:teuthology.orchestra.run.smithi149.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 1.3 MB/s | 49 kB 00:00 2023-07-20T18:56:49.888 INFO:teuthology.orchestra.run.smithi149.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2023-07-20T18:56:49.930 INFO:teuthology.orchestra.run.smithi149.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-07-20T18:56:49.997 INFO:teuthology.orchestra.run.smithi149.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.2 MB/s | 283 kB 00:00 2023-07-20T18:56:50.009 INFO:teuthology.orchestra.run.smithi061.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 89 kB/s | 18 kB 00:00 2023-07-20T18:56:50.038 INFO:teuthology.orchestra.run.smithi149.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 374 kB/s | 16 kB 00:00 2023-07-20T18:56:50.089 INFO:teuthology.orchestra.run.smithi149.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 673 kB/s | 34 kB 00:00 2023-07-20T18:56:50.122 INFO:teuthology.orchestra.run.smithi149.stdout:(57/131): python3-kubernetes-11.0.0-6.el8.noarc 3.3 MB/s | 1.2 MB 00:00 2023-07-20T18:56:50.173 INFO:teuthology.orchestra.run.smithi149.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2023-07-20T18:56:50.198 INFO:teuthology.orchestra.run.smithi149.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 818 kB/s | 61 kB 00:00 2023-07-20T18:56:50.232 INFO:teuthology.orchestra.run.smithi149.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 319 kB/s | 19 kB 00:00 2023-07-20T18:56:50.265 INFO:teuthology.orchestra.run.smithi149.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 358 kB/s | 24 kB 00:00 2023-07-20T18:56:50.291 INFO:teuthology.orchestra.run.smithi149.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 499 kB/s | 29 kB 00:00 2023-07-20T18:56:50.324 INFO:teuthology.orchestra.run.smithi149.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 471 kB/s | 27 kB 00:00 2023-07-20T18:56:50.426 INFO:teuthology.orchestra.run.smithi061.stdout:(85/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 4.4 MB/s | 48 MB 00:10 2023-07-20T18:56:50.449 INFO:teuthology.orchestra.run.smithi149.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-07-20T18:56:50.460 INFO:teuthology.orchestra.run.smithi061.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 528 kB/s | 238 kB 00:00 2023-07-20T18:56:50.475 INFO:teuthology.orchestra.run.smithi149.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.4 MB/s | 255 kB 00:00 2023-07-20T18:56:50.502 INFO:teuthology.orchestra.run.smithi061.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 906 kB/s | 605 kB 00:00 2023-07-20T18:56:50.533 INFO:teuthology.orchestra.run.smithi149.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 734 kB/s | 61 kB 00:00 2023-07-20T18:56:50.559 INFO:teuthology.orchestra.run.smithi149.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-07-20T18:56:50.601 INFO:teuthology.orchestra.run.smithi149.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2023-07-20T18:56:50.603 INFO:teuthology.orchestra.run.smithi061.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 537 kB/s | 72 kB 00:00 2023-07-20T18:56:50.626 INFO:teuthology.orchestra.run.smithi149.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T18:56:50.776 INFO:teuthology.orchestra.run.smithi149.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-07-20T18:56:50.893 INFO:teuthology.orchestra.run.smithi149.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.7 MB/s | 200 kB 00:00 2023-07-20T18:56:50.919 INFO:teuthology.orchestra.run.smithi061.stdout:(89/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 4.7 MB/s | 2.3 MB 00:00 2023-07-20T18:56:50.945 INFO:teuthology.orchestra.run.smithi061.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 176 kB/s | 60 kB 00:00 2023-07-20T18:56:50.970 INFO:teuthology.orchestra.run.smithi061.stdout:(91/131): fuse-2.9.7-15.el8.x86_64.rpm 177 kB/s | 83 kB 00:00 2023-07-20T18:56:51.035 INFO:teuthology.orchestra.run.smithi149.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 3.6 MB/s | 1.7 MB 00:00 2023-07-20T18:56:51.061 INFO:teuthology.orchestra.run.smithi149.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2023-07-20T18:56:51.078 INFO:teuthology.orchestra.run.smithi061.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 356 kB/s | 39 kB 00:00 2023-07-20T18:56:51.103 INFO:teuthology.orchestra.run.smithi061.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 363 kB/s | 58 kB 00:00 2023-07-20T18:56:51.129 INFO:teuthology.orchestra.run.smithi061.stdout:(94/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 469 kB/s | 98 kB 00:00 2023-07-20T18:56:51.161 INFO:teuthology.orchestra.run.smithi149.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 509 kB/s | 48 kB 00:00 2023-07-20T18:56:51.194 INFO:teuthology.orchestra.run.smithi149.stdout:(75/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2023-07-20T18:56:51.204 INFO:teuthology.orchestra.run.smithi061.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.0 MB/s | 644 kB 00:00 2023-07-20T18:56:51.246 INFO:teuthology.orchestra.run.smithi061.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.2 MB/s | 171 kB 00:00 2023-07-20T18:56:51.271 INFO:teuthology.orchestra.run.smithi061.stdout:(97/131): NetworkManager-initscripts-updown-1.3 968 kB/s | 138 kB 00:00 2023-07-20T18:56:51.286 INFO:teuthology.orchestra.run.smithi149.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 695 kB/s | 87 kB 00:00 2023-07-20T18:56:51.345 INFO:teuthology.orchestra.run.smithi149.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 284 kB/s | 43 kB 00:00 2023-07-20T18:56:51.363 INFO:teuthology.orchestra.run.smithi061.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 591 kB/s | 54 kB 00:00 2023-07-20T18:56:51.414 INFO:teuthology.orchestra.run.smithi061.stdout:(99/131): python3-pyasn1-modules-0.3.7-6.el8.no 658 kB/s | 110 kB 00:00 2023-07-20T18:56:51.420 INFO:teuthology.orchestra.run.smithi149.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 817 kB/s | 109 kB 00:00 2023-07-20T18:56:51.446 INFO:teuthology.orchestra.run.smithi149.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2023-07-20T18:56:51.497 INFO:teuthology.orchestra.run.smithi061.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2023-07-20T18:56:51.554 INFO:teuthology.orchestra.run.smithi149.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 173 kB/s | 19 kB 00:00 2023-07-20T18:56:51.556 INFO:teuthology.orchestra.run.smithi061.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2023-07-20T18:56:51.588 INFO:teuthology.orchestra.run.smithi149.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.3 MB/s | 560 kB 00:00 2023-07-20T18:56:51.640 INFO:teuthology.orchestra.run.smithi061.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 273 kB/s | 39 kB 00:00 2023-07-20T18:56:51.705 INFO:teuthology.orchestra.run.smithi149.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 4.7 MB/s | 559 kB 00:00 2023-07-20T18:56:51.730 INFO:teuthology.orchestra.run.smithi149.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 883 kB/s | 155 kB 00:00 2023-07-20T18:56:51.790 INFO:teuthology.orchestra.run.smithi061.stdout:(103/131): python3-scipy-1.0.0-21.module+el8.5. 23 MB/s | 14 MB 00:00 2023-07-20T18:56:51.824 INFO:teuthology.orchestra.run.smithi061.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.9 MB/s | 538 kB 00:00 2023-07-20T18:56:51.839 INFO:teuthology.orchestra.run.smithi149.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 134 kB/s | 18 kB 00:00 2023-07-20T18:56:51.857 INFO:teuthology.orchestra.run.smithi061.stdout:(105/131): bison-3.0.4-10.el8.x86_64.rpm 2.2 MB/s | 688 kB 00:00 2023-07-20T18:56:51.892 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-07-20T18:56:51.908 INFO:teuthology.orchestra.run.smithi149.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.3 MB/s | 605 kB 00:00 2023-07-20T18:56:51.924 INFO:teuthology.orchestra.run.smithi061.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-07-20T18:56:51.949 INFO:teuthology.orchestra.run.smithi061.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-07-20T18:56:51.973 INFO:teuthology.orchestra.run.smithi149.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2023-07-20T18:56:52.000 INFO:teuthology.orchestra.run.smithi061.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.9 MB/s | 713 kB 00:00 2023-07-20T18:56:52.082 INFO:teuthology.orchestra.run.smithi149.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 2.3 MB 00:00 2023-07-20T18:56:52.104 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T18:56:52.132 INFO:teuthology.orchestra.run.smithi149.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 456 kB/s | 72 kB 00:00 2023-07-20T18:56:52.150 INFO:teuthology.orchestra.run.smithi061.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-07-20T18:56:52.175 INFO:teuthology.orchestra.run.smithi061.stdout:(110/131): python3-jsonpatch-1.21-2.el8.noarch. 118 kB/s | 27 kB 00:00 2023-07-20T18:56:52.182 INFO:teuthology.orchestra.run.smithi149.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 825 kB/s | 83 kB 00:00 2023-07-20T18:56:52.200 INFO:teuthology.orchestra.run.smithi061.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2023-07-20T18:56:52.280 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T18:56:52.282 INFO:teuthology.orchestra.run.smithi149.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 399 kB/s | 60 kB 00:00 2023-07-20T18:56:52.309 INFO:teuthology.orchestra.run.smithi061.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 277 kB/s | 44 kB 00:00 2023-07-20T18:56:52.333 INFO:teuthology.orchestra.run.smithi149.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 653 kB/s | 98 kB 00:00 2023-07-20T18:56:52.342 INFO:teuthology.orchestra.run.smithi061.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.2 MB/s | 457 kB 00:00 2023-07-20T18:56:52.416 INFO:teuthology.orchestra.run.smithi149.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 431 kB/s | 58 kB 00:00 2023-07-20T18:56:52.423 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:56:52.435 INFO:teuthology.orchestra.run.smithi061.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2023-07-20T18:56:52.460 INFO:teuthology.orchestra.run.smithi061.stdout:(115/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2023-07-20T18:56:52.466 INFO:teuthology.orchestra.run.smithi149.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 289 kB/s | 39 kB 00:00 2023-07-20T18:56:52.477 INFO:teuthology.orchestra.run.smithi061.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 478 kB/s | 64 kB 00:00 2023-07-20T18:56:52.494 INFO:teuthology.orchestra.run.smithi061.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.1 MB/s | 126 kB 00:00 2023-07-20T18:56:52.535 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:56:52.550 INFO:teuthology.orchestra.run.smithi149.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 4.7 MB/s | 644 kB 00:00 2023-07-20T18:56:52.575 INFO:teuthology.orchestra.run.smithi149.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.5 MB/s | 171 kB 00:00 2023-07-20T18:56:52.611 INFO:teuthology.orchestra.run.smithi061.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 133 kB/s | 20 kB 00:00 2023-07-20T18:56:52.661 INFO:teuthology.orchestra.run.smithi061.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 563 kB/s | 103 kB 00:00 2023-07-20T18:56:52.694 INFO:teuthology.orchestra.run.smithi061.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2023-07-20T18:56:52.701 INFO:teuthology.orchestra.run.smithi149.stdout:(96/131): NetworkManager-initscripts-updown-1.3 917 kB/s | 138 kB 00:00 2023-07-20T18:56:52.770 INFO:teuthology.orchestra.run.smithi061.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2023-07-20T18:56:52.773 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:56:52.784 INFO:teuthology.orchestra.run.smithi149.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.3 MB/s | 110 kB 00:00 2023-07-20T18:56:52.851 INFO:teuthology.orchestra.run.smithi149.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 814 kB/s | 54 kB 00:00 2023-07-20T18:56:52.853 INFO:teuthology.orchestra.run.smithi061.stdout:(122/131): openblas-threads-0.3.15-3.el8.x86_64 20 MB/s | 4.9 MB 00:00 2023-07-20T18:56:52.879 INFO:teuthology.orchestra.run.smithi061.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.8 MB/s | 202 kB 00:00 2023-07-20T18:56:52.987 INFO:teuthology.orchestra.run.smithi061.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-20T18:56:53.038 INFO:teuthology.orchestra.run.smithi061.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.9 MB/s | 354 kB 00:00 2023-07-20T18:56:53.043 INFO:teuthology.orchestra.run.smithi149.stdout:(99/131): python3-scipy-1.0.0-21.module+el8.5.0 29 MB/s | 14 MB 00:00 2023-07-20T18:56:53.068 INFO:teuthology.orchestra.run.smithi149.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-07-20T18:56:53.138 INFO:teuthology.orchestra.run.smithi061.stdout:(126/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-07-20T18:56:53.171 INFO:teuthology.orchestra.run.smithi061.stdout:(127/131): postgresql-contrib-10.21-2.module+el 4.3 MB/s | 808 kB 00:00 2023-07-20T18:56:53.178 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:56:53.194 INFO:teuthology.orchestra.run.smithi149.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2023-07-20T18:56:53.222 INFO:teuthology.orchestra.run.smithi061.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 8.2 MB/s | 1.5 MB 00:00 2023-07-20T18:56:53.261 INFO:teuthology.orchestra.run.smithi149.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 202 kB/s | 39 kB 00:00 2023-07-20T18:56:53.361 INFO:teuthology.orchestra.run.smithi149.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 4.0 MB/s | 688 kB 00:00 2023-07-20T18:56:53.372 INFO:teuthology.orchestra.run.smithi061.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.2 MB/s | 798 kB 00:00 2023-07-20T18:56:53.398 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:56:53.453 INFO:teuthology.orchestra.run.smithi149.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.7 MB/s | 538 kB 00:00 2023-07-20T18:56:53.479 INFO:teuthology.orchestra.run.smithi149.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2023-07-20T18:56:53.620 INFO:teuthology.orchestra.run.smithi149.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 4.9 MB/s | 713 kB 00:00 2023-07-20T18:56:53.646 INFO:teuthology.orchestra.run.smithi149.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 975 kB/s | 187 kB 00:00 2023-07-20T18:56:53.746 INFO:teuthology.orchestra.run.smithi149.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 265 kB/s | 27 kB 00:00 2023-07-20T18:56:53.826 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:56:53.855 INFO:teuthology.orchestra.run.smithi149.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-20T18:56:53.888 INFO:teuthology.orchestra.run.smithi149.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.2 MB/s | 320 kB 00:00 2023-07-20T18:56:53.897 INFO:teuthology.orchestra.run.smithi061.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 5.0 MB/s | 3.6 MB 00:00 2023-07-20T18:56:53.980 INFO:teuthology.orchestra.run.smithi149.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 352 kB/s | 44 kB 00:00 2023-07-20T18:56:54.039 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:56:54.072 INFO:teuthology.orchestra.run.smithi149.stdout:(112/131): python3-numpy-1.14.3-10.el8.x86_64.r 20 MB/s | 3.7 MB 00:00 2023-07-20T18:56:54.106 INFO:teuthology.orchestra.run.smithi149.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.6 MB/s | 457 kB 00:00 2023-07-20T18:56:54.107 INFO:teuthology.orchestra.run.smithi061.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 3.8 MB/s | 3.7 MB 00:00 2023-07-20T18:56:54.108 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:56:54.108 INFO:teuthology.orchestra.run.smithi061.stdout:Total 9.3 MB/s | 206 MB 00:22 2023-07-20T18:56:54.164 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:56:54.181 INFO:teuthology.orchestra.run.smithi149.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.0 MB/s | 225 kB 00:00 2023-07-20T18:56:54.256 INFO:teuthology.orchestra.run.smithi149.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 426 kB/s | 64 kB 00:00 2023-07-20T18:56:54.281 INFO:teuthology.orchestra.run.smithi149.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.2 MB/s | 126 kB 00:00 2023-07-20T18:56:54.349 INFO:teuthology.orchestra.run.smithi149.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 217 kB/s | 20 kB 00:00 2023-07-20T18:56:54.365 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:56:54.407 INFO:teuthology.orchestra.run.smithi149.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 826 kB/s | 103 kB 00:00 2023-07-20T18:56:54.474 INFO:teuthology.orchestra.run.smithi149.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2023-07-20T18:56:54.478 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:56:54.636 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:56:54.649 INFO:teuthology.orchestra.run.smithi149.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 20 MB/s | 4.9 MB 00:00 2023-07-20T18:56:54.693 INFO:teuthology.orchestra.run.smithi149.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.0 MB/s | 198 kB 00:00 2023-07-20T18:56:54.792 INFO:teuthology.orchestra.run.smithi149.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 1.6 MB/s | 202 kB 00:00 2023-07-20T18:56:54.799 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T18:56:54.964 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-07-20T18:56:54.984 INFO:teuthology.orchestra.run.smithi149.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-07-20T18:56:55.027 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T18:56:55.051 INFO:teuthology.orchestra.run.smithi149.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.3 MB/s | 354 kB 00:00 2023-07-20T18:56:55.096 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-07-20T18:56:55.096 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-07-20T18:56:55.126 INFO:teuthology.orchestra.run.smithi149.stdout:(125/131): postgresql-contrib-10.21-2.module+el 11 MB/s | 808 kB 00:00 2023-07-20T18:56:55.201 INFO:teuthology.orchestra.run.smithi149.stdout:(126/131): postgresql-server-10.21-2.module+el8 23 MB/s | 5.1 MB 00:00 2023-07-20T18:56:55.285 INFO:teuthology.orchestra.run.smithi149.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 9.5 MB/s | 1.5 MB 00:00 2023-07-20T18:56:55.478 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T18:56:55.859 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T18:56:56.093 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T18:56:56.177 INFO:teuthology.orchestra.run.smithi149.stdout:(128/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 4.1 MB/s | 3.6 MB 00:00 2023-07-20T18:56:56.231 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:56:56.252 INFO:teuthology.orchestra.run.smithi149.stdout:(129/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 3.5 MB/s | 3.7 MB 00:01 2023-07-20T18:56:56.302 INFO:teuthology.orchestra.run.smithi149.stdout:(130/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 6.2 MB/s | 798 kB 00:00 2023-07-20T18:56:56.330 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-07-20T18:56:56.334 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-07-20T18:56:56.359 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:56:56.506 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:56:56.652 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:56:56.901 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T18:56:57.070 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T18:56:57.232 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T18:56:57.253 INFO:teuthology.orchestra.run.smithi149.stdout:(131/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 3.6 MB/s | 48 MB 00:13 2023-07-20T18:56:57.257 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:56:57.258 INFO:teuthology.orchestra.run.smithi149.stdout:Total 8.3 MB/s | 206 MB 00:24 2023-07-20T18:56:57.389 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:56:57.535 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:56:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:56:58.075 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-07-20T18:56:58.196 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-07-20T18:56:58.196 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-07-20T18:56:58.319 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:56:58.489 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:56:59.077 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:56:59.264 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:56:59.325 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-07-20T18:56:59.329 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-07-20T18:56:59.957 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:57:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T18:57:00.386 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T18:57:00.640 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T18:57:00.862 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T18:57:01.035 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T18:57:01.259 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T18:57:01.423 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-07-20T18:57:01.472 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:57:01.585 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T18:57:01.770 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T18:57:02.036 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:57:02.139 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:57:02.361 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:57:02.776 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:57:03.003 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:57:03.227 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:57:03.421 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:57:03.728 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:57:03.860 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:57:04.067 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:57:04.171 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:57:04.247 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T18:57:04.333 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:57:04.388 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-07-20T18:57:04.513 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T18:57:04.562 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T18:57:04.576 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T18:57:04.743 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T18:57:04.785 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T18:57:04.866 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T18:57:04.936 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:57:04.982 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T18:57:05.049 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T18:57:05.157 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T18:57:05.253 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:57:05.379 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T18:57:05.392 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T18:57:05.565 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T18:57:05.624 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T18:57:05.684 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T18:57:05.769 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:57:05.777 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T18:57:05.878 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:57:05.919 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T18:57:06.036 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:57:06.044 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:57:06.062 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T18:57:06.190 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:57:06.234 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T18:57:06.396 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:57:06.407 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T18:57:06.449 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T18:57:06.461 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:57:06.585 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T18:57:06.627 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:57:06.707 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T18:57:06.752 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:57:06.778 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T18:57:06.933 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T18:57:06.961 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:57:06.961 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:57:07.057 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:57:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:57:07.121 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:57:07.199 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:57:07.215 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:57:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:57:07.386 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T18:57:07.392 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T18:57:07.575 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T18:57:07.607 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T18:57:07.737 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T18:57:07.814 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T18:57:07.897 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T18:57:07.938 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:57:08.090 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T18:57:08.136 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:57:08.185 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T18:57:08.241 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T18:57:08.439 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T18:57:08.469 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T18:57:08.586 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:57:08.675 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T18:57:08.695 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T18:57:08.769 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:57:08.852 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:57:08.952 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:57:08.998 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T18:57:09.079 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T18:57:09.223 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T18:57:09.392 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T18:57:09.423 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T18:57:09.569 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T18:57:09.628 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T18:57:09.710 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:57:09.734 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:57:09.850 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T18:57:09.852 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T18:57:09.984 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T18:57:10.010 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:57:10.092 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T18:57:10.204 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T18:57:10.429 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T18:57:10.490 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T18:57:10.633 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T18:57:10.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T18:57:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T18:57:10.794 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:57:10.823 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T18:57:10.874 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T18:57:11.063 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T18:57:11.101 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T18:57:11.277 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:57:11.300 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T18:57:11.375 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T18:57:11.474 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T18:57:11.552 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:57:11.761 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T18:57:12.009 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T18:57:12.236 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T18:57:12.259 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T18:57:12.467 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T18:57:12.480 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T18:57:12.650 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T18:57:12.670 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T18:57:12.848 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T18:57:12.904 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T18:57:13.067 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T18:57:13.106 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T18:57:13.142 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:57:13.188 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T18:57:13.328 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T18:57:13.407 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T18:57:13.466 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T18:57:13.618 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:57:13.647 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T18:57:13.829 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T18:57:14.049 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T18:57:14.205 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T18:57:14.222 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T18:57:14.373 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T18:57:14.515 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T18:57:14.535 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T18:57:14.759 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T18:57:14.808 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T18:57:15.017 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T18:57:15.038 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T18:57:15.201 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T18:57:15.248 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T18:57:15.403 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T18:57:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T18:57:15.473 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T18:57:15.650 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T18:57:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T18:57:15.817 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T18:57:15.878 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T18:57:15.976 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T18:57:16.097 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T18:57:16.160 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T18:57:16.318 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:57:16.385 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:57:16.425 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T18:57:16.614 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T18:57:16.726 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T18:57:16.864 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T18:57:16.972 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T18:57:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:57:17.096 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:57:17.213 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T18:57:17.289 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T18:57:17.325 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T18:57:17.447 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T18:57:17.506 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T18:57:17.540 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T18:57:17.637 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T18:57:17.667 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T18:57:17.820 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T18:57:17.823 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T18:57:17.828 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T18:57:17.974 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T18:57:17.987 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T18:57:18.112 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T18:57:18.125 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T18:57:18.131 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T18:57:18.273 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T18:57:18.360 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T18:57:18.427 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T18:57:18.434 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:57:18.506 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T18:57:18.558 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T18:57:18.661 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T18:57:18.738 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T18:57:18.931 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T18:57:18.938 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T18:57:19.093 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:57:19.176 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T18:57:19.212 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T18:57:19.388 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T18:57:19.429 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T18:57:19.447 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T18:57:19.546 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T18:57:19.611 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:57:19.690 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:57:19.708 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T18:57:19.711 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T18:57:19.845 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T18:57:19.877 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T18:57:19.917 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T18:57:20.036 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T18:57:20.162 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T18:57:20.253 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T18:57:20.374 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T18:57:20.476 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T18:57:20.556 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T18:57:20.675 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T18:57:20.910 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T18:57:20.923 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T18:57:21.117 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T18:57:21.228 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T18:57:21.299 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T18:57:21.534 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T18:57:21.627 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T18:57:21.782 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T18:57:21.930 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T18:57:22.031 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T18:57:22.193 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T18:57:22.320 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T18:57:22.500 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T18:57:22.679 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T18:57:22.804 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T18:57:22.914 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T18:57:22.961 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T18:57:23.071 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T18:57:23.191 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T18:57:23.278 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T18:57:23.463 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T18:57:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T18:57:23.587 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:23.693 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T18:57:23.828 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:23.866 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T18:57:23.889 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T18:57:24.064 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T18:57:24.089 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T18:57:24.256 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T18:57:24.294 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T18:57:24.410 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:24.441 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T18:57:24.466 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T18:57:24.612 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T18:57:24.665 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T18:57:24.704 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:57:24.783 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:57:24.854 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T18:57:24.893 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T18:57:24.911 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:57:24.999 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T18:57:25.107 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T18:57:25.254 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T18:57:25.258 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T18:57:25.393 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T18:57:25.498 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T18:57:25.599 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T18:57:25.699 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T18:57:25.802 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T18:57:25.953 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T18:57:25.977 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T18:57:26.170 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T18:57:26.176 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T18:57:26.306 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T18:57:26.359 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T18:57:26.527 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T18:57:26.580 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T18:57:26.745 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T18:57:26.922 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T18:57:27.163 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T18:57:27.357 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T18:57:27.514 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T18:57:27.738 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T18:57:27.748 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T18:57:28.082 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T18:57:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T18:57:28.645 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T18:57:28.856 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T18:57:28.936 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T18:57:29.113 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T18:57:29.122 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T18:57:29.392 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T18:57:29.687 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T18:57:29.951 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T18:57:29.980 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T18:57:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:57:30.179 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T18:57:30.220 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:57:30.384 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T18:57:30.962 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T18:57:31.116 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:57:31.194 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T18:57:31.408 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T18:57:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:34.584 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:35.108 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:35.208 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:35.324 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:35.466 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:57:35.540 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:57:35.691 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:57:35.923 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T18:57:36.192 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:57:36.271 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T18:57:36.746 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:57:44.254 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:57:44.254 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2023-07-20T18:57:44.254 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /run 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2023-07-20T18:57:44.255 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-07-20T18:57:44.256 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:57:44.617 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:57:46.203 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:57:46.375 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:57:46.595 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:57:46.766 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:57:47.221 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:57:47.400 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:57:47.417 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:57:49.091 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:57:49.319 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:57:49.438 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T18:57:49.630 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T18:57:49.849 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:57:50.067 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:57:50.374 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:57:51.338 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:57:51.482 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:57:51.573 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:57:51.945 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T18:57:52.186 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:57:52.222 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:57:52.365 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:57:52.377 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:57:53.723 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T18:57:53.940 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:57:53.974 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:57:54.115 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:57:54.158 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:57:54.299 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:57:54.651 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:57:54.840 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:57:55.182 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:57:55.182 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /sys 2023-07-20T18:57:55.182 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /proc 2023-07-20T18:57:55.182 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /dev 2023-07-20T18:57:55.182 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /run 2023-07-20T18:57:55.183 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /mnt 2023-07-20T18:57:55.183 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /var/tmp 2023-07-20T18:57:55.183 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /home 2023-07-20T18:57:55.183 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /tmp 2023-07-20T18:57:55.183 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /dev 2023-07-20T18:57:55.183 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:57:55.539 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:57:55.642 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T18:57:55.642 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /sys 2023-07-20T18:57:55.642 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /proc 2023-07-20T18:57:55.642 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2023-07-20T18:57:55.643 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /run 2023-07-20T18:57:55.643 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /mnt 2023-07-20T18:57:55.643 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /var/tmp 2023-07-20T18:57:55.643 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /home 2023-07-20T18:57:55.643 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /tmp 2023-07-20T18:57:55.643 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2023-07-20T18:57:55.644 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:57:55.963 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:57:57.238 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:57:57.414 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:57:57.550 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T18:57:57.645 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:57:57.739 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:57:57.864 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:57:57.958 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T18:57:58.138 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:57:58.350 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:57:58.523 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:57:58.541 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:57:58.593 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T18:57:58.738 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:57:58.755 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T18:57:59.413 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:58:00.280 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:58:00.310 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T18:58:00.441 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:58:00.466 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:58:00.613 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T18:58:00.629 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T18:58:00.683 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T18:58:00.809 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T18:58:00.818 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T18:58:00.846 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T18:58:00.995 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T18:58:01.031 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:58:01.046 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T18:58:01.179 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:58:01.241 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:58:01.254 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:58:01.395 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T18:58:01.433 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:58:01.563 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:58:01.660 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T18:58:01.672 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:58:01.843 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T18:58:02.007 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T18:58:02.589 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:58:02.637 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T18:58:02.796 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:58:02.812 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:58:02.893 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:58:02.904 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T18:58:03.234 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T18:58:03.285 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T18:58:03.451 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:58:03.488 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:58:03.489 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:58:03.525 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T18:58:03.622 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:58:03.633 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:58:03.687 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:58:03.698 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T18:58:04.179 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:04.455 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:04.473 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:04.632 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T18:58:04.963 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T18:58:05.004 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T18:58:05.017 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T18:58:05.201 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:58:05.213 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:58:05.236 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:58:05.237 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T18:58:05.248 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T18:58:05.395 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:58:05.397 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:58:05.437 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:58:05.440 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T18:58:05.586 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T18:58:05.587 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:58:05.639 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:58:05.780 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:58:05.871 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:58:05.939 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:58:06.013 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T18:58:06.072 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:58:06.085 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:58:06.087 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:58:06.189 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:58:06.230 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:58:06.238 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:58:06.374 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T18:58:06.374 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:06.575 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:06.575 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:58:06.835 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:58:07.037 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:07.549 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:07.549 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T18:58:07.549 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T18:58:07.549 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T18:58:07.550 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T18:58:07.551 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T18:58:07.551 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T18:58:07.551 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T18:58:07.551 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T18:58:07.551 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T18:58:07.551 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T18:58:07.552 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T18:58:07.552 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T18:58:07.552 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T18:58:07.552 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T18:58:07.552 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T18:58:07.552 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T18:58:07.553 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T18:58:07.553 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T18:58:07.560 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T18:58:07.560 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T18:58:07.560 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T18:58:07.560 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T18:58:07.560 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T18:58:07.560 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T18:58:07.561 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T18:58:07.561 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T18:58:07.561 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T18:58:07.561 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T18:58:07.561 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T18:58:07.562 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T18:58:07.563 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T18:58:07.564 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T18:58:07.565 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T18:58:07.566 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T18:58:07.567 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T18:58:07.567 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T18:58:07.567 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T18:58:07.567 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T18:58:07.567 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T18:58:07.568 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T18:58:07.568 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T18:58:07.568 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T18:58:07.568 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T18:58:07.568 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T18:58:07.568 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T18:58:07.569 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T18:58:07.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T18:58:07.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T18:58:07.572 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T18:58:07.572 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T18:58:07.572 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T18:58:07.572 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T18:58:07.573 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T18:58:07.573 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T18:58:07.573 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T18:58:07.573 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T18:58:07.574 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T18:58:07.574 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T18:58:07.574 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T18:58:07.574 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T18:58:07.574 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T18:58:07.575 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T18:58:07.575 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T18:58:07.575 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T18:58:07.575 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T18:58:07.576 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T18:58:07.576 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T18:58:07.576 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T18:58:07.576 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T18:58:07.576 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T18:58:07.577 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T18:58:07.577 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T18:58:07.577 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:07.577 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T18:58:10.045 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:10.045 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:52 ago on Thu 20 Jul 2023 06:56:17 PM UTC. 2023-07-20T18:58:10.668 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:58:11.068 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T18:58:11.569 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T18:58:11.876 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T18:58:12.035 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T18:58:12.047 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T18:58:12.284 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T18:58:12.333 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T18:58:12.484 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:58:12.516 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T18:58:12.688 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:58:12.788 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T18:58:12.923 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T18:58:12.937 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:58:13.082 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T18:58:13.143 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T18:58:13.229 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T18:58:13.376 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T18:58:13.517 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T18:58:13.664 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T18:58:15.410 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:15.652 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:15.670 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:15.819 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T18:58:15.958 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:16.093 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T18:58:16.229 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:16.248 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T18:58:16.388 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T18:58:16.422 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T18:58:16.679 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T18:58:16.752 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T18:58:16.935 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:58:17.017 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:58:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T18:58:17.210 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:58:17.223 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:58:17.350 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T18:58:17.385 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:58:17.393 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:58:17.479 INFO:teuthology.orchestra.run.smithi156.stdout:Installed products updated. 2023-07-20T18:58:17.529 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T18:58:17.529 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:17.570 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:58:17.653 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T18:58:17.693 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:58:17.693 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2023-07-20T18:58:17.693 INFO:teuthology.orchestra.run.smithi156.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T18:58:17.694 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.696 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.697 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-3.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T18:58:17.698 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T18:58:17.699 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T18:58:17.700 INFO:teuthology.orchestra.run.smithi156.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T18:58:17.701 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:17.702 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.703 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T18:58:17.704 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T18:58:17.705 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T18:58:17.706 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T18:58:17.707 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T18:58:17.708 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T18:58:17.709 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T18:58:17.710 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T18:58:17.711 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T18:58:17.711 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T18:58:17.711 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T18:58:17.711 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T18:58:17.711 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-07-20T18:58:17.730 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:17.730 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:58:17.821 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:58:17.833 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T18:58:17.898 DEBUG:teuthology.parallel:result is None 2023-07-20T18:58:17.949 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:58:17.970 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:58:17.979 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T18:58:18.097 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T18:58:18.098 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:18.158 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:18.332 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:18.332 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:58:18.576 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T18:58:18.683 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:18.683 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T18:58:18.683 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T18:58:18.683 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T18:58:18.684 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T18:58:18.685 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T18:58:18.686 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T18:58:18.687 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T18:58:18.687 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T18:58:18.687 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T18:58:18.687 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T18:58:18.694 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T18:58:18.694 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T18:58:18.694 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T18:58:18.695 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T18:58:18.696 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T18:58:18.697 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T18:58:18.698 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T18:58:18.699 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T18:58:18.700 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T18:58:18.701 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T18:58:18.702 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T18:58:18.703 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T18:58:18.704 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T18:58:18.705 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T18:58:18.706 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T18:58:18.707 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:18.708 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T18:58:18.761 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:19.297 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:19.297 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T18:58:19.297 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T18:58:19.298 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T18:58:19.299 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T18:58:19.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T18:58:19.301 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T18:58:19.301 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T18:58:19.301 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T18:58:19.301 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T18:58:19.301 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T18:58:19.301 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T18:58:19.302 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T18:58:19.302 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T18:58:19.302 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T18:58:19.302 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T18:58:19.302 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T18:58:19.303 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T18:58:19.304 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T18:58:19.305 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T18:58:19.305 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T18:58:19.305 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T18:58:19.305 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T18:58:19.305 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T18:58:19.305 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T18:58:19.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T18:58:19.307 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T18:58:19.308 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T18:58:19.309 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T18:58:19.310 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T18:58:19.311 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T18:58:19.312 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T18:58:19.312 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T18:58:19.312 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T18:58:19.312 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T18:58:19.313 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T18:58:19.313 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T18:58:19.313 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T18:58:19.313 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T18:58:19.314 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T18:58:19.314 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T18:58:19.314 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T18:58:19.314 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T18:58:19.314 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T18:58:19.315 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T18:58:19.316 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T18:58:19.317 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T18:58:19.317 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T18:58:19.317 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T18:58:21.169 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:21.169 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:01 ago on Thu 20 Jul 2023 06:56:20 PM UTC. 2023-07-20T18:58:21.761 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T18:58:21.761 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:02 ago on Thu 20 Jul 2023 06:56:19 PM UTC. 2023-07-20T18:58:30.818 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-07-20T18:58:31.036 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:58:31.037 INFO:teuthology.orchestra.run.smithi149.stdout:Upgraded: 2023-07-20T18:58:31.037 INFO:teuthology.orchestra.run.smithi149.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T18:58:31.037 INFO:teuthology.orchestra.run.smithi149.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.037 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.037 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.038 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.039 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.040 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T18:58:31.041 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: jq-1.6-3.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.042 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.043 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.044 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T18:58:31.044 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.044 INFO:teuthology.orchestra.run.smithi149.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T18:58:31.044 INFO:teuthology.orchestra.run.smithi149.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T18:58:31.044 INFO:teuthology.orchestra.run.smithi149.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T18:58:31.044 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T18:58:31.045 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T18:58:31.046 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T18:58:31.047 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T18:58:31.047 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T18:58:31.047 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T18:58:31.047 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.047 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.047 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T18:58:31.048 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T18:58:31.049 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T18:58:31.050 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T18:58:31.051 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.052 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T18:58:31.053 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.054 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.055 INFO:teuthology.orchestra.run.smithi149.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T18:58:31.055 INFO:teuthology.orchestra.run.smithi149.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T18:58:31.055 INFO:teuthology.orchestra.run.smithi149.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T18:58:31.055 INFO:teuthology.orchestra.run.smithi149.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T18:58:31.055 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T18:58:31.056 INFO:teuthology.orchestra.run.smithi149.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T18:58:31.056 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-07-20T18:58:31.056 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-07-20T18:58:31.496 DEBUG:teuthology.parallel:result is None 2023-07-20T18:58:31.572 INFO:teuthology.orchestra.run.smithi061.stdout:Installed products updated. 2023-07-20T18:58:31.805 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:58:31.805 INFO:teuthology.orchestra.run.smithi061.stdout:Upgraded: 2023-07-20T18:58:31.805 INFO:teuthology.orchestra.run.smithi061.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T18:58:31.806 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.807 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.808 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.808 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.808 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.808 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.808 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.808 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.809 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.809 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.809 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.809 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.809 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.809 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.810 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.810 INFO:teuthology.orchestra.run.smithi061.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.810 INFO:teuthology.orchestra.run.smithi061.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T18:58:31.810 INFO:teuthology.orchestra.run.smithi061.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T18:58:31.810 INFO:teuthology.orchestra.run.smithi061.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T18:58:31.810 INFO:teuthology.orchestra.run.smithi061.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: jq-1.6-3.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.811 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T18:58:31.812 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T18:58:31.813 INFO:teuthology.orchestra.run.smithi061.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T18:58:31.814 INFO:teuthology.orchestra.run.smithi061.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T18:58:31.815 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T18:58:31.816 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T18:58:31.817 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T18:58:31.818 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T18:58:31.819 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T18:58:31.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T18:58:31.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.822 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.823 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T18:58:31.823 INFO:teuthology.orchestra.run.smithi061.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T18:58:31.823 INFO:teuthology.orchestra.run.smithi061.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T18:58:31.823 INFO:teuthology.orchestra.run.smithi061.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T18:58:31.823 INFO:teuthology.orchestra.run.smithi061.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T18:58:31.823 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T18:58:31.824 INFO:teuthology.orchestra.run.smithi061.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T18:58:31.824 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:58:31.824 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-07-20T18:58:32.007 DEBUG:teuthology.parallel:result is None 2023-07-20T18:58:32.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:58:37.280 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T18:58:37.309 INFO:teuthology.orchestra.run.smithi061.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:58:37.312 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:58:37.312 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T18:58:37.313 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:58:42.545 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T18:58:42.573 INFO:teuthology.orchestra.run.smithi149.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:58:42.573 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:58:42.574 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T18:58:42.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:58:46.711 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T18:58:46.739 INFO:teuthology.orchestra.run.smithi156.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:58:46.740 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T18:58:46.740 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T18:58:46.741 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T18:58:46.742 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:58:46.742 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T18:58:46.794 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T18:58:46.794 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T18:58:46.842 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T18:58:46.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T18:58:46.890 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T18:58:46.891 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:58:46.891 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T18:58:46.923 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T18:58:46.993 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T18:58:46.994 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T18:58:47.024 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T18:58:47.093 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T18:58:47.093 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T18:58:47.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T18:58:47.196 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T18:58:47.196 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:58:47.196 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T18:58:47.227 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T18:58:47.297 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T18:58:47.325 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T18:58:47.353 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T18:58:47.420 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T18:58:47.421 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T18:58:47.450 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T18:58:47.519 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T18:58:47.519 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:58:47.520 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T18:58:47.550 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T18:58:47.620 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T18:58:47.621 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T18:58:47.650 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T18:58:47.718 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T18:58:47.719 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T18:58:47.748 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T18:58:47.817 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T18:58:47.914 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-07-20T18:58:47.915 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:58:47.926 INFO:tasks.cephadm:Cluster fsid is 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:58:47.927 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T18:58:47.927 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.61', 'mon.b': '172.21.15.149', 'mon.c': '172.21.15.156'} 2023-07-20T18:58:47.928 INFO:tasks.cephadm:First mon is mon.a on smithi061 2023-07-20T18:58:47.928 INFO:tasks.cephadm:First mgr is x 2023-07-20T18:58:47.928 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T18:58:47.929 DEBUG:teuthology.orchestra.run.smithi061:> sudo hostname $(hostname -s) 2023-07-20T18:58:47.964 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname $(hostname -s) 2023-07-20T18:58:47.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2023-07-20T18:58:48.024 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T18:58:48.025 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:58:48.305 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T18:58:48.558 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T18:58:48.559 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T18:58:48.559 DEBUG:teuthology.orchestra.run.smithi061:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:49.338 INFO:teuthology.orchestra.run.smithi061.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 18:58 /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:49.338 DEBUG:teuthology.orchestra.run.smithi149:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.095 INFO:teuthology.orchestra.run.smithi149.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 18:58 /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.096 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.841 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 18:58 /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.841 DEBUG:teuthology.orchestra.run.smithi061:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.864 DEBUG:teuthology.orchestra.run.smithi149:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.885 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T18:58:50.922 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T18:58:50.922 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T18:58:50.925 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T18:58:50.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T18:58:52.279 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:58:52.286 INFO:teuthology.orchestra.run.smithi149.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:58:52.311 INFO:teuthology.orchestra.run.smithi061.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:59:28.616 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2023-07-20T18:59:28.617 INFO:teuthology.orchestra.run.smithi061.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:59:28.617 INFO:teuthology.orchestra.run.smithi061.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T18:59:28.617 INFO:teuthology.orchestra.run.smithi061.stdout: "repo_digests": [ 2023-07-20T18:59:28.617 INFO:teuthology.orchestra.run.smithi061.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T18:59:28.618 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2023-07-20T18:59:28.618 INFO:teuthology.orchestra.run.smithi061.stdout:} 2023-07-20T18:59:36.746 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2023-07-20T18:59:36.746 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:59:36.746 INFO:teuthology.orchestra.run.smithi149.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T18:59:36.747 INFO:teuthology.orchestra.run.smithi149.stdout: "repo_digests": [ 2023-07-20T18:59:36.747 INFO:teuthology.orchestra.run.smithi149.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T18:59:36.747 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2023-07-20T18:59:36.747 INFO:teuthology.orchestra.run.smithi149.stdout:} 2023-07-20T18:59:39.449 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-07-20T18:59:39.449 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T18:59:39.450 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T18:59:39.450 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2023-07-20T18:59:39.450 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T18:59:39.450 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-07-20T18:59:39.450 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-07-20T18:59:39.469 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph 2023-07-20T18:59:39.509 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph 2023-07-20T18:59:39.546 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2023-07-20T18:59:39.576 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /etc/ceph 2023-07-20T18:59:39.607 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /etc/ceph 2023-07-20T18:59:39.637 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2023-07-20T18:59:39.663 INFO:tasks.cephadm:Writing seed config... 2023-07-20T18:59:39.664 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T18:59:39.665 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T18:59:39.665 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T18:59:39.665 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T18:59:39.665 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T18:59:39.666 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-20T18:59:39.666 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T18:59:39.666 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T18:59:39.666 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T18:59:39.666 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T18:59:39.666 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T18:59:39.667 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T18:59:39.667 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T18:59:39.667 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T18:59:39.667 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T18:59:39.667 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T18:59:39.668 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T18:59:39.668 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T18:59:39.668 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T18:59:39.668 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T18:59:39.668 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T18:59:39.669 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-20T18:59:39.669 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T18:59:39.669 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T18:59:39.669 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T18:59:39.670 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T18:59:39.670 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T18:59:39.670 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T18:59:39.670 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T18:59:39.670 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-07-20T18:59:39.671 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T18:59:39.671 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-20T18:59:39.671 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-20T18:59:39.671 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T18:59:39.671 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T18:59:39.672 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-20T18:59:39.672 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T18:59:39.672 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T18:59:39.673 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T18:59:39.673 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T18:59:39.674 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T18:59:39.674 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T18:59:39.674 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T18:59:39.675 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-20T18:59:39.677 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T18:59:39.677 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T18:59:39.697 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7541eb30-272f-11ee-9b34-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T18:59:39.698 DEBUG:teuthology.orchestra.run.smithi061:mon.a> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T18:59:39.740 DEBUG:teuthology.orchestra.run.smithi061:mgr.x> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T18:59:39.769 INFO:journalctl@ceph.mon.a.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T18:59:39.770 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T18:59:39.770 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.61 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:59:39.798 INFO:journalctl@ceph.mgr.x.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T18:59:39.940 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-07-20T18:59:39.941 INFO:teuthology.orchestra.run.smithi061.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '7541eb30-272f-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.61', '--skip-admin-label'] 2023-07-20T18:59:40.092 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stdout 4.1.1 2023-07-20T18:59:40.093 INFO:teuthology.orchestra.run.smithi061.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T18:59:40.093 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying podman|docker is present... 2023-07-20T18:59:40.240 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stdout 4.1.1 2023-07-20T18:59:40.241 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying lvm2 is present... 2023-07-20T18:59:40.241 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying time synchronization is in place... 2023-07-20T18:59:40.247 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T18:59:40.247 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T18:59:40.253 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T18:59:40.253 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout inactive 2023-07-20T18:59:40.259 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout enabled 2023-07-20T18:59:40.265 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout active 2023-07-20T18:59:40.265 INFO:teuthology.orchestra.run.smithi061.stdout:Unit chronyd.service is enabled and running 2023-07-20T18:59:40.265 INFO:teuthology.orchestra.run.smithi061.stdout:Repeating the final host check... 2023-07-20T18:59:40.411 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stdout 4.1.1 2023-07-20T18:59:40.411 INFO:teuthology.orchestra.run.smithi061.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T18:59:40.411 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl is present 2023-07-20T18:59:40.411 INFO:teuthology.orchestra.run.smithi061.stdout:lvcreate is present 2023-07-20T18:59:40.417 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T18:59:40.417 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T18:59:40.423 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T18:59:40.423 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout inactive 2023-07-20T18:59:40.429 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout enabled 2023-07-20T18:59:40.435 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout active 2023-07-20T18:59:40.435 INFO:teuthology.orchestra.run.smithi061.stdout:Unit chronyd.service is enabled and running 2023-07-20T18:59:40.435 INFO:teuthology.orchestra.run.smithi061.stdout:Host looks OK 2023-07-20T18:59:40.435 INFO:teuthology.orchestra.run.smithi061.stdout:Cluster fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:40.435 INFO:teuthology.orchestra.run.smithi061.stdout:Acquiring lock 140360459377128 on /run/cephadm/7541eb30-272f-11ee-9b34-001a4aab830c.lock 2023-07-20T18:59:40.436 INFO:teuthology.orchestra.run.smithi061.stdout:Lock 140360459377128 acquired on /run/cephadm/7541eb30-272f-11ee-9b34-001a4aab830c.lock 2023-07-20T18:59:40.436 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying IP 172.21.15.61 port 3300 ... 2023-07-20T18:59:40.436 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying IP 172.21.15.61 port 6789 ... 2023-07-20T18:59:40.437 INFO:teuthology.orchestra.run.smithi061.stdout:Base mon IP(s) is [172.21.15.61:3300, 172.21.15.61:6789], mon addrv is [v2:172.21.15.61:3300,v1:172.21.15.61:6789] 2023-07-20T18:59:40.440 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-20T18:59:40.440 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.61 metric 100 2023-07-20T18:59:40.443 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T18:59:40.443 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-20T18:59:40.443 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-20T18:59:40.445 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T18:59:40.445 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T18:59:40.446 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T18:59:40.446 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-20T18:59:40.446 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout inet6 fe80::d486:e2fe:b046:d421/64 scope link noprefixroute 2023-07-20T18:59:40.446 INFO:teuthology.orchestra.run.smithi061.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T18:59:40.446 INFO:teuthology.orchestra.run.smithi061.stdout:Mon IP `172.21.15.61` is in CIDR network `172.21.0.0/20` 2023-07-20T18:59:40.447 INFO:teuthology.orchestra.run.smithi061.stdout:Mon IP `172.21.15.61` is in CIDR network `172.21.0.0/20` 2023-07-20T18:59:40.447 INFO:teuthology.orchestra.run.smithi061.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T18:59:40.447 INFO:teuthology.orchestra.run.smithi061.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T18:59:40.448 INFO:teuthology.orchestra.run.smithi061.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:59:40.893 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T18:59:40.894 INFO:teuthology.orchestra.run.smithi061.stdout:/bin/podman: stderr Storing signatures 2023-07-20T18:59:41.746 INFO:teuthology.orchestra.run.smithi061.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T18:59:41.747 INFO:teuthology.orchestra.run.smithi061.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T18:59:41.747 INFO:teuthology.orchestra.run.smithi061.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T18:59:42.398 INFO:teuthology.orchestra.run.smithi061.stdout:stat: stdout 167 167 2023-07-20T18:59:42.398 INFO:teuthology.orchestra.run.smithi061.stdout:Creating initial keys... 2023-07-20T18:59:43.249 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph-authtool: stdout AQAehLlkf+nyKhAAaHf0YplcVU4CfrIGdlTpNg== 2023-07-20T18:59:44.001 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph-authtool: stdout AQAfhLlkNAV/IRAAqeA3kKsUwcsM116y1NPh0g== 2023-07-20T18:59:44.757 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph-authtool: stdout AQAghLlky06VExAAwfzO6M3g4InS4EqEWDxV0Q== 2023-07-20T18:59:44.757 INFO:teuthology.orchestra.run.smithi061.stdout:Creating initial monmap... 2023-07-20T18:59:45.606 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T18:59:45.606 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T18:59:45.607 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:45.607 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T18:59:45.607 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool for a [v2:172.21.15.61:3300,v1:172.21.15.61:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T18:59:45.607 INFO:teuthology.orchestra.run.smithi061.stdout:setting min_mon_release = pacific 2023-07-20T18:59:45.607 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/monmaptool: set fsid to 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:45.607 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T18:59:45.608 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T18:59:45.608 INFO:teuthology.orchestra.run.smithi061.stdout:Creating mon... 2023-07-20T18:59:46.358 INFO:teuthology.orchestra.run.smithi061.stdout:create mon.a on 2023-07-20T18:59:46.495 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T18:59:46.587 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T18:59:46.724 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7541eb30-272f-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-7541eb30-272f-11ee-9b34-001a4aab830c.target. 2023-07-20T18:59:46.724 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7541eb30-272f-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-7541eb30-272f-11ee-9b34-001a4aab830c.target. 2023-07-20T18:59:46.901 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a 2023-07-20T18:59:46.902 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T18:59:46.997 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7541eb30-272f-11ee-9b34-001a4aab830c.target.wants/ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-7541eb30-272f-11ee-9b34-001a4aab830c@.service. 2023-07-20T18:59:47.310 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:46 smithi061 systemd[1]: Starting Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T18:59:47.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:47 smithi061 podman[126907]: 2023-07-20T18:59:47.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:47 smithi061 bash[126907]: 761b31532ac165a385d7e87cfbbf96be08a91b05767e21f742d06b313458375a 2023-07-20T18:59:47.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:47 smithi061 ceph-mon[126946]: mkfs 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:47.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:47 smithi061 ceph-mon[126946]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T18:59:47.604 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:59:47.605 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout disabled 2023-07-20T18:59:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:59:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout inactive 2023-07-20T18:59:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:firewalld.service is not enabled 2023-07-20T18:59:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T18:59:47.613 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for mon to start... 2023-07-20T18:59:47.613 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for mon... 2023-07-20T18:59:47.860 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:47 smithi061 systemd[1]: Started Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T18:59:48.660 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T18:59:48.665 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout id: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:48.665 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T18:59:48.665 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T18:59:48.665 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout services: 2023-07-20T18:59:48.665 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.674528s) 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout data: 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T18:59:48.666 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.029+0000 7fd7b2486700 1 Processor -- start 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.029+0000 7fd7b2486700 1 -- start start 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.029+0000 7fd7b2486700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac106860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.029+0000 7fd7b2486700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac106e30 con 0x7fd7ac106440 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:48.667 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57130/0 (socket says 172.21.15.61:57130) 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7abfff700 1 -- 172.21.15.61:0/72275348 learned_addr learned my addr 172.21.15.61:0/72275348 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7abfff700 1 -- 172.21.15.61:0/72275348 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac106f70 con 0x7fd7ac106440 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7abfff700 1 --2- 172.21.15.61:0/72275348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac106860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00bb40 tx=0x7fd79c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe39e8be9edadb1f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7ab7fe700 1 -- 172.21.15.61:0/72275348 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd79c00dbe0 con 0x7fd7ac106440 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7ab7fe700 1 -- 172.21.15.61:0/72275348 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c00dd40 con 0x7fd7ac106440 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.030+0000 7fd7ab7fe700 1 -- 172.21.15.61:0/72275348 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd79c004030 con 0x7fd7ac106440 2023-07-20T18:59:48.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 -- 172.21.15.61:0/72275348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 msgr2=0x7fd7ac106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 --2- 172.21.15.61:0/72275348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac106860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00bb40 tx=0x7fd79c00be50 comp rx=0 tx=0).stop 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 -- 172.21.15.61:0/72275348 shutdown_connections 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 --2- 172.21.15.61:0/72275348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac106860 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 -- 172.21.15.61:0/72275348 >> 172.21.15.61:0/72275348 conn(0x7fd7ac0ffbc0 msgr2=0x7fd7ac101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 -- 172.21.15.61:0/72275348 shutdown_connections 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 -- 172.21.15.61:0/72275348 wait complete. 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 Processor -- start 2023-07-20T18:59:48.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.031+0000 7fd7b2486700 1 -- start start 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7b2486700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7b2486700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac197750 con 0x7fd7ac106440 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57132/0 (socket says 172.21.15.61:57132) 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7abfff700 1 -- 172.21.15.61:0/3939068147 learned_addr learned my addr 172.21.15.61:0/3939068147 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7abfff700 1 -- 172.21.15.61:0/3939068147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c00b7f0 con 0x7fd7ac106440 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7abfff700 1 --2- 172.21.15.61:0/3939068147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac197210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00b340 tx=0x7fd79c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd79c020070 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c003d40 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac197950 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd79c010470 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.032+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac197d70 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.033+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd79c0108d0 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.033+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd79c02e9e0 con 0x7fd7ac106440 2023-07-20T18:59:48.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.034+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac190ac0 con 0x7fd7ac106440 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.036+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd79c029070 con 0x7fd7ac106440 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.168+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd7ac02c970 con 0x7fd7ac106440 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.169+0000 7fd7a9ffb700 1 -- 172.21.15.61:0/3939068147 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fd7ac106f70 con 0x7fd7ac106440 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.169+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 msgr2=0x7fd7ac197210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.169+0000 7fd7b2486700 1 --2- 172.21.15.61:0/3939068147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac197210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00b340 tx=0x7fd79c004750 comp rx=0 tx=0).stop 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.169+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 shutdown_connections 2023-07-20T18:59:48.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.169+0000 7fd7b2486700 1 --2- 172.21.15.61:0/3939068147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7ac106440 0x7fd7ac197210 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:48.673 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.169+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 >> 172.21.15.61:0/3939068147 conn(0x7fd7ac0ffbc0 msgr2=0x7fd7ac101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:48.673 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.170+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 shutdown_connections 2023-07-20T18:59:48.673 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:48.170+0000 7fd7b2486700 1 -- 172.21.15.61:0/3939068147 wait complete. 2023-07-20T18:59:48.673 INFO:teuthology.orchestra.run.smithi061.stdout:mon is available 2023-07-20T18:59:48.673 INFO:teuthology.orchestra.run.smithi061.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T18:59:48.823 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:48 smithi061 ceph-mon[126946]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T18:59:48.824 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:48 smithi061 ceph-mon[126946]: monmap e1: 1 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0]} removed_ranks: {} 2023-07-20T18:59:48.824 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:48 smithi061 ceph-mon[126946]: fsmap 2023-07-20T18:59:48.824 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:48 smithi061 ceph-mon[126946]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T18:59:48.825 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:48 smithi061 ceph-mon[126946]: mgrmap e1: no daemons active 2023-07-20T18:59:48.825 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:48 smithi061 ceph-mon[126946]: from='client.? 172.21.15.61:0/3939068147' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T18:59:49.666 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:49 smithi061 ceph-mon[126946]: from='client.? 172.21.15.61:0/3263786377' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T18:59:49.666 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:49 smithi061 ceph-mon[126946]: from='client.? 172.21.15.61:0/3263786377' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T18:59:49.708 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T18:59:49.708 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout fsid = 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.61:3300,v1:172.21.15.61:6789] 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T18:59:49.709 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T18:59:49.710 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T18:59:49.713 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T18:59:49.714 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931ebec700 1 Processor -- start 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931ebec700 1 -- start start 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931ebec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f9318104c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931ebec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318079e70 con 0x7f93181047f0 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931c988700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f9318104c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931c988700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f9318104c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57134/0 (socket says 172.21.15.61:57134) 2023-07-20T18:59:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931c988700 1 -- 172.21.15.61:0/1706518197 learned_addr learned my addr 172.21.15.61:0/1706518197 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.186+0000 7f931c988700 1 -- 172.21.15.61:0/1706518197 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318105150 con 0x7f93181047f0 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931c988700 1 --2- 172.21.15.61:0/1706518197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f9318104c10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f930c0098a0 tx=0x7f930c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c4f4129dd6cd83 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f93177fe700 1 -- 172.21.15.61:0/1706518197 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f930c004160 con 0x7f93181047f0 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f93177fe700 1 -- 172.21.15.61:0/1706518197 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c0042c0 con 0x7f93181047f0 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f93177fe700 1 -- 172.21.15.61:0/1706518197 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f930c0045a0 con 0x7f93181047f0 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 -- 172.21.15.61:0/1706518197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 msgr2=0x7f9318104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 --2- 172.21.15.61:0/1706518197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f9318104c10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f930c0098a0 tx=0x7f930c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:59:49.716 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 -- 172.21.15.61:0/1706518197 shutdown_connections 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 --2- 172.21.15.61:0/1706518197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f9318104c10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 -- 172.21.15.61:0/1706518197 >> 172.21.15.61:0/1706518197 conn(0x7f93181003f0 msgr2=0x7f9318102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 -- 172.21.15.61:0/1706518197 shutdown_connections 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.187+0000 7f931ebec700 1 -- 172.21.15.61:0/1706518197 wait complete. 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931ebec700 1 Processor -- start 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931ebec700 1 -- start start 2023-07-20T18:59:49.717 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931ebec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f931819b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931ebec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931819bb00 con 0x7f93181047f0 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931c988700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f931819b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931c988700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f931819b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57136/0 (socket says 172.21.15.61:57136) 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.188+0000 7f931c988700 1 -- 172.21.15.61:0/3263786377 learned_addr learned my addr 172.21.15.61:0/3263786377 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f931c988700 1 -- 172.21.15.61:0/3263786377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c009550 con 0x7f93181047f0 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f931c988700 1 --2- 172.21.15.61:0/3263786377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f931819b5c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f93180792b0 tx=0x7f930c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f930c011630 con 0x7f93181047f0 2023-07-20T18:59:49.718 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c011790 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931819bd00 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f930c011a30 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931819c120 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f930c022070 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.189+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f930c01c450 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.190+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9318195280 con 0x7f93181047f0 2023-07-20T18:59:49.719 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.193+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f930c01b450 con 0x7f93181047f0 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.318+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f931802cd80 con 0x7f93181047f0 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.322+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f930c01b5f0 con 0x7f93181047f0 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.322+0000 7f9315ffb700 1 -- 172.21.15.61:0/3263786377 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f930c024630 con 0x7f93181047f0 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 msgr2=0x7f931819b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 --2- 172.21.15.61:0/3263786377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f931819b5c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f93180792b0 tx=0x7f930c004750 comp rx=0 tx=0).stop 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 shutdown_connections 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 --2- 172.21.15.61:0/3263786377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93181047f0 0x7f931819b5c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:49.720 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 >> 172.21.15.61:0/3263786377 conn(0x7f93181003f0 msgr2=0x7f93181923c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:49.721 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 shutdown_connections 2023-07-20T18:59:49.721 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:49.323+0000 7f931ebec700 1 -- 172.21.15.61:0/3263786377 wait complete. 2023-07-20T18:59:49.721 INFO:teuthology.orchestra.run.smithi061.stdout:Generating new minimal ceph.conf... 2023-07-20T18:59:50.792 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:50 smithi061 ceph-mon[126946]: from='client.? 172.21.15.61:0/472724230' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T18:59:50.858 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.260+0000 7f5127c5a700 1 Processor -- start 2023-07-20T18:59:50.858 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.260+0000 7f5127c5a700 1 -- start start 2023-07-20T18:59:50.858 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f5127c5a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f51201056f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:50.858 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f5127c5a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5120105cc0 con 0x7f51201052d0 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51259f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f51201056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51259f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f51201056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57138/0 (socket says 172.21.15.61:57138) 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51259f6700 1 -- 172.21.15.61:0/3969345745 learned_addr learned my addr 172.21.15.61:0/3969345745 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51259f6700 1 -- 172.21.15.61:0/3969345745 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5120105ff0 con 0x7f51201052d0 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51259f6700 1 --2- 172.21.15.61:0/3969345745 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f51201056f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f511c0098a0 tx=0x7f511c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda5cd862a1d12b2 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51249f4700 1 -- 172.21.15.61:0/3969345745 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f511c004160 con 0x7f51201052d0 2023-07-20T18:59:50.859 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51249f4700 1 -- 172.21.15.61:0/3969345745 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f511c0042c0 con 0x7f51201052d0 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.261+0000 7f51249f4700 1 -- 172.21.15.61:0/3969345745 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f511c011a00 con 0x7f51201052d0 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 -- 172.21.15.61:0/3969345745 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 msgr2=0x7f51201056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 --2- 172.21.15.61:0/3969345745 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f51201056f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f511c0098a0 tx=0x7f511c009bb0 comp rx=0 tx=0).stop 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 -- 172.21.15.61:0/3969345745 shutdown_connections 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 --2- 172.21.15.61:0/3969345745 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f51201056f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 -- 172.21.15.61:0/3969345745 >> 172.21.15.61:0/3969345745 conn(0x7f51200ffb50 msgr2=0x7f5120101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:50.860 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 -- 172.21.15.61:0/3969345745 shutdown_connections 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.262+0000 7f5127c5a700 1 -- 172.21.15.61:0/3969345745 wait complete. 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f5127c5a700 1 Processor -- start 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f5127c5a700 1 -- start start 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f5127c5a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f512019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f5127c5a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512019c270 con 0x7f51201052d0 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f51259f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f512019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f51259f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f512019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57140/0 (socket says 172.21.15.61:57140) 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f51259f6700 1 -- 172.21.15.61:0/472724230 learned_addr learned my addr 172.21.15.61:0/472724230 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.263+0000 7f51259f6700 1 -- 172.21.15.61:0/472724230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c009550 con 0x7f51201052d0 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f51259f6700 1 --2- 172.21.15.61:0/472724230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f512019bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f511c009fb0 tx=0x7f511c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f511c003930 con 0x7f51201052d0 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f511c003a90 con 0x7f51201052d0 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f511c019610 con 0x7f51201052d0 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512019c470 con 0x7f51201052d0 2023-07-20T18:59:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512019c890 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f511c019bd0 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.264+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f511c0243f0 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.265+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5120061da0 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.267+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f511c003c00 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.393+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5120109380 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.394+0000 7f5112ffd700 1 -- 172.21.15.61:0/472724230 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f5120109380 con 0x7f51201052d0 2023-07-20T18:59:50.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 msgr2=0x7f512019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 --2- 172.21.15.61:0/472724230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f512019bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f511c009fb0 tx=0x7f511c011ad0 comp rx=0 tx=0).stop 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 shutdown_connections 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 --2- 172.21.15.61:0/472724230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51201052d0 0x7f512019bd30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 >> 172.21.15.61:0/472724230 conn(0x7f51200ffb50 msgr2=0x7f51201046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 shutdown_connections 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:50.395+0000 7f5127c5a700 1 -- 172.21.15.61:0/472724230 wait complete. 2023-07-20T18:59:50.864 INFO:teuthology.orchestra.run.smithi061.stdout:Restarting the monitor... 2023-07-20T18:59:51.193 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:50 smithi061 systemd[1]: Stopping Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T18:59:51.462 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-a[126922]: 2023-07-20T18:59:51.140+0000 7f9450e18700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T18:59:51.462 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-a[126922]: 2023-07-20T18:59:51.140+0000 7f9450e18700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T18:59:51.962 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:51 smithi061 bash[127284]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-a 2023-07-20T18:59:52.300 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:51 smithi061 systemd[1]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T18:59:52.301 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:51 smithi061 systemd[1]: Stopped Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T18:59:52.301 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:51 smithi061 systemd[1]: Starting Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T18:59:52.502 INFO:teuthology.orchestra.run.smithi061.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T18:59:52.555 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 podman[127414]: 2023-07-20T18:59:52.556 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 bash[127414]: a086a33e920fbee7016ba11dcb4306c85ac51a174e69a64db5e57a10654529be 2023-07-20T18:59:52.556 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T18:59:52.556 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T18:59:52.556 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: pidfile_write: ignore empty --pid-file 2023-07-20T18:59:52.556 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: load: jerasure load: lrc 2023-07-20T18:59:52.556 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: RocksDB version: 7.9.2 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Git sha 0 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: DB SUMMARY 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: DB Session ID: 590DRBQMGVONL92A0VSM 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: CURRENT file: CURRENT 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T18:59:52.557 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98689 ; 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.error_if_exists: 0 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.create_if_missing: 0 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.paranoid_checks: 1 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T18:59:52.558 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.env: 0x557d32ffdc60 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.info_log: 0x557d34da12c0 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.statistics: (nil) 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.use_fsync: 0 2023-07-20T18:59:52.559 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_log_file_size: 0 2023-07-20T18:59:52.560 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T18:59:52.560 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T18:59:52.560 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T18:59:52.560 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T18:59:52.560 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.allow_fallocate: 1 2023-07-20T18:59:52.560 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.use_direct_reads: 0 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.db_log_dir: 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.wal_dir: 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T18:59:52.561 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T18:59:52.562 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T18:59:52.562 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T18:59:52.562 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T18:59:52.562 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T18:59:52.562 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T18:59:52.562 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T18:59:52.563 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.write_buffer_manager: 0x557d34cc23c0 2023-07-20T18:59:52.563 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T18:59:52.563 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T18:59:52.563 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T18:59:52.563 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T18:59:52.563 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.unordered_write: 0 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T18:59:52.564 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.row_cache: None 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.wal_filter: None 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.two_write_queues: 0 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T18:59:52.565 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.wal_compression: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.atomic_flush: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.log_readahead_size: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T18:59:52.566 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_background_jobs: 2 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_background_compactions: -1 2023-07-20T18:59:52.567 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_subcompactions: 1 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T18:59:52.568 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_open_files: -1 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_background_flushes: -1 2023-07-20T18:59:52.569 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Compression algorithms supported: 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kZSTD supported: 0 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kXpressCompression supported: 0 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kZlibCompression supported: 1 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kSnappyCompression supported: 1 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kLZ4Compression supported: 1 2023-07-20T18:59:52.570 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: kBZip2Compression supported: 0 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.merge_operator: 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_filter: None 2023-07-20T18:59:52.571 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557d34da13c0) 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout: cache_index_and_filter_blocks: 1 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T18:59:52.572 INFO:journalctl@ceph.mon.a.smithi061.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: pin_top_level_index_and_filter: 1 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: index_type: 0 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: data_block_index_type: 0 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: index_shortening: 1 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: checksum: 4 2023-07-20T18:59:52.573 INFO:journalctl@ceph.mon.a.smithi061.stdout: no_block_cache: 0 2023-07-20T18:59:52.574 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_cache: 0x557d34d61350 2023-07-20T18:59:52.574 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_cache_name: BinnedLRUCache 2023-07-20T18:59:52.574 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_cache_options: 2023-07-20T18:59:52.574 INFO:journalctl@ceph.mon.a.smithi061.stdout: capacity : 536870912 2023-07-20T18:59:52.574 INFO:journalctl@ceph.mon.a.smithi061.stdout: num_shard_bits : 4 2023-07-20T18:59:52.574 INFO:journalctl@ceph.mon.a.smithi061.stdout: strict_capacity_limit : 0 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: high_pri_pool_ratio: 0.000 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_cache_compressed: (nil) 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: persistent_cache: (nil) 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_size: 4096 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_size_deviation: 10 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_restart_interval: 16 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: index_block_restart_interval: 1 2023-07-20T18:59:52.575 INFO:journalctl@ceph.mon.a.smithi061.stdout: metadata_block_size: 4096 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: partition_filters: 0 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: use_delta_encoding: 1 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: filter_policy: bloomfilter 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: whole_key_filtering: 1 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: verify_compression: 0 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: read_amp_bytes_per_bit: 0 2023-07-20T18:59:52.576 INFO:journalctl@ceph.mon.a.smithi061.stdout: format_version: 5 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout: enable_index_compression: 1 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout: block_align: 0 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout: max_auto_readahead_size: 262144 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout: prepopulate_block_cache: 0 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout: initial_auto_readahead_size: 8192 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T18:59:52.577 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression: NoCompression 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.num_levels: 7 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T18:59:52.578 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T18:59:52.579 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T18:59:52.580 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T18:59:52.581 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T18:59:52.582 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T18:59:52.582 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T18:59:52.582 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T18:59:52.582 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T18:59:52.582 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T18:59:52.582 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T18:59:52.583 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T18:59:52.583 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T18:59:52.583 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T18:59:52.583 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T18:59:52.584 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T18:59:52.584 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T18:59:52.584 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T18:59:52.584 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T18:59:52.585 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T18:59:52.585 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T18:59:52.585 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T18:59:52.585 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T18:59:52.586 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T18:59:52.586 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T18:59:52.586 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T18:59:52.586 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T18:59:52.587 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T18:59:52.587 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T18:59:52.587 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T18:59:52.587 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T18:59:52.588 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T18:59:52.588 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T18:59:52.588 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.table_properties_collectors: 2023-07-20T18:59:52.589 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.inplace_update_support: 0 2023-07-20T18:59:52.589 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T18:59:52.589 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T18:59:52.590 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T18:59:52.590 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T18:59:52.590 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.bloom_locality: 0 2023-07-20T18:59:52.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.max_successive_merges: 0 2023-07-20T18:59:52.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T18:59:52.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T18:59:52.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T18:59:52.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T18:59:52.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.ttl: 2592000 2023-07-20T18:59:52.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T18:59:52.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T18:59:52.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T18:59:52.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.enable_blob_files: false 2023-07-20T18:59:52.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.min_blob_size: 0 2023-07-20T18:59:52.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T18:59:52.593 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T18:59:52.594 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T18:59:52.594 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T18:59:52.594 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T18:59:52.594 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T18:59:52.595 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T18:59:52.595 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T18:59:52.595 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T18:59:52.596 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T18:59:52.596 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 018355cc-d22a-4707-bf29-fc3f455e7f65 2023-07-20T18:59:52.596 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879592428238, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T18:59:52.596 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T18:59:52.596 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879592429318, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93930, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91935, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 84500, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689879592, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "018355cc-d22a-4707-bf29-fc3f455e7f65", "db_session_id": "590DRBQMGVONL92A0VSM", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T18:59:52.596 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879592429411, "job": 1, "event": "recovery_finished"} 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557d34dfa000 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: DB pointer 0x557d34de6000 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T18:59:52.597 INFO:journalctl@ceph.mon.a.smithi061.stdout: ** DB Stats ** 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T18:59:52.598 INFO:journalctl@ceph.mon.a.smithi061.stdout: 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: ** Compaction Stats [default] ** 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: L0 2/0 93.56 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: Sum 2/0 93.56 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: 2023-07-20T18:59:52.599 INFO:journalctl@ceph.mon.a.smithi061.stdout: ** Compaction Stats [default] ** 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: 2023-07-20T18:59:52.600 INFO:journalctl@ceph.mon.a.smithi061.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: Cumulative compaction: 0.00 GB write, 25.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: Interval compaction: 0.00 GB write, 25.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T18:59:52.601 INFO:journalctl@ceph.mon.a.smithi061.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout: Block cache BinnedLRUCache@0x557d34d61350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout: 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: starting mon.a rank 0 at public addrs [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] at bind addrs [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???) e1 preinit fsid 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).mds e1 new map 2023-07-20T18:59:52.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).mds e1 print_map 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout: e1 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout: legacy client fscid: -1 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout: 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout: No filesystems configured 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T18:59:52.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expand map: {default=false} 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta from 'false' to 'false' 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expanded map: {default=false} 2023-07-20T18:59:52.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expand map: {default=info} 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta from 'info' to 'info' 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expanded map: {default=info} 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expand map: {default=daemon} 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expand map: {default=debug} 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T18:59:52.605 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: expand_channel_meta expanded map: {default=debug} 2023-07-20T18:59:52.606 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 systemd[1]: Started Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T18:59:52.606 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T18:59:52.606 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: monmap e1: 1 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0]} removed_ranks: {} 2023-07-20T18:59:52.606 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: fsmap 2023-07-20T18:59:52.606 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T18:59:52.606 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:52 smithi061 ceph-mon[127454]: mgrmap e1: no daemons active 2023-07-20T18:59:54.463 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 18:59:54 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1788382947' entity='client.admin' 2023-07-20T18:59:54.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.928+0000 7f1b0978e700 1 Processor -- start 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.928+0000 7f1b0978e700 1 -- start start 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.928+0000 7f1b0978e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0407a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.928+0000 7f1b0978e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0407a910 con 0x7f1b0407bee0 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0407a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0407a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57142/0 (socket says 172.21.15.61:57142) 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b02ffd700 1 -- 172.21.15.61:0/436088865 learned_addr learned my addr 172.21.15.61:0/436088865 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:54.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b02ffd700 1 -- 172.21.15.61:0/436088865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0407aa50 con 0x7f1b0407bee0 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b02ffd700 1 --2- 172.21.15.61:0/436088865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0407a340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1af40098a0 tx=0x7f1af4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0c55e15586fcf9 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b01ffb700 1 -- 172.21.15.61:0/436088865 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1af4004160 con 0x7f1b0407bee0 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.929+0000 7f1b01ffb700 1 -- 172.21.15.61:0/436088865 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1af40042c0 con 0x7f1b0407bee0 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b01ffb700 1 -- 172.21.15.61:0/436088865 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1af4011a00 con 0x7f1b0407bee0 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 -- 172.21.15.61:0/436088865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 msgr2=0x7f1b0407a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 --2- 172.21.15.61:0/436088865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0407a340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1af40098a0 tx=0x7f1af4009bb0 comp rx=0 tx=0).stop 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 -- 172.21.15.61:0/436088865 shutdown_connections 2023-07-20T18:59:54.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 --2- 172.21.15.61:0/436088865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0407a340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 -- 172.21.15.61:0/436088865 >> 172.21.15.61:0/436088865 conn(0x7f1b04101550 msgr2=0x7f1b04103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 -- 172.21.15.61:0/436088865 shutdown_connections 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.930+0000 7f1b0978e700 1 -- 172.21.15.61:0/436088865 wait complete. 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b0978e700 1 Processor -- start 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b0978e700 1 -- start start 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b0978e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b0978e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0419c320 con 0x7f1b0407bee0 2023-07-20T18:59:54.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0419bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57144/0 (socket says 172.21.15.61:57144) 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.931+0000 7f1b02ffd700 1 -- 172.21.15.61:0/1788382947 learned_addr learned my addr 172.21.15.61:0/1788382947 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1b02ffd700 1 -- 172.21.15.61:0/1788382947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af4009550 con 0x7f1b0407bee0 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1b02ffd700 1 --2- 172.21.15.61:0/1788382947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0419bde0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1af4006af0 tx=0x7f1af4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1af4003a40 con 0x7f1b0407bee0 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1af4003ba0 con 0x7f1b0407bee0 2023-07-20T18:59:54.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1af401a700 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1b0978e700 1 -- 172.21.15.61:0/1788382947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0419c520 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1b0978e700 1 -- 172.21.15.61:0/1788382947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0419c940 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1af401ac80 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.932+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1af4019700 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.933+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae80052f0 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:52.936+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f1af4024ad0 con 0x7f1b0407bee0 2023-07-20T18:59:54.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.061+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f1ae8005c90 con 0x7f1b0407bee0 2023-07-20T18:59:54.562 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.063+0000 7f1afbfff700 1 -- 172.21.15.61:0/1788382947 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1af4024c70 con 0x7f1b0407bee0 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 msgr2=0x7f1b0419bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 --2- 172.21.15.61:0/1788382947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0419bde0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1af4006af0 tx=0x7f1af4011ad0 comp rx=0 tx=0).stop 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 shutdown_connections 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 --2- 172.21.15.61:0/1788382947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b0407bee0 0x7f1b0419bde0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 >> 172.21.15.61:0/1788382947 conn(0x7f1b04101550 msgr2=0x7f1b04102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 shutdown_connections 2023-07-20T18:59:54.563 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T18:59:53.064+0000 7f1af9ffb700 1 -- 172.21.15.61:0/1788382947 wait complete. 2023-07-20T18:59:54.564 INFO:teuthology.orchestra.run.smithi061.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T18:59:54.641 INFO:teuthology.orchestra.run.smithi061.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T18:59:54.642 INFO:teuthology.orchestra.run.smithi061.stdout:Creating mgr... 2023-07-20T18:59:54.642 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying port 9283 ... 2023-07-20T18:59:54.643 INFO:teuthology.orchestra.run.smithi061.stdout:Verifying port 8765 ... 2023-07-20T18:59:55.001 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T18:59:55.001 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T18:59:55.100 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7541eb30-272f-11ee-9b34-001a4aab830c.target.wants/ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-7541eb30-272f-11ee-9b34-001a4aab830c@.service. 2023-07-20T18:59:55.213 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:55 smithi061 systemd[1]: Starting Ceph mgr.x for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T18:59:56.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:56 smithi061 podman[127688]: 2023-07-20T18:59:58.218 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:59:58.219 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout disabled 2023-07-20T18:59:58.226 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:59:58.227 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout inactive 2023-07-20T18:59:58.227 INFO:teuthology.orchestra.run.smithi061.stdout:firewalld.service is not enabled 2023-07-20T18:59:58.227 INFO:teuthology.orchestra.run.smithi061.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T18:59:58.235 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T18:59:58.235 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout disabled 2023-07-20T18:59:58.243 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T18:59:58.243 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout inactive 2023-07-20T18:59:58.243 INFO:teuthology.orchestra.run.smithi061.stdout:firewalld.service is not enabled 2023-07-20T18:59:58.243 INFO:teuthology.orchestra.run.smithi061.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T18:59:58.243 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for mgr to start... 2023-07-20T18:59:58.243 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for mgr... 2023-07-20T18:59:58.462 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 bash[127688]: 5ba8b8647fce0c9d47cee62720f15a3d81b58f59392046f400885951447fa086 2023-07-20T18:59:58.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:58.143+0000 7f650d683200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T18:59:58.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 systemd[1]: Started Ceph mgr.x for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T18:59:58.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:58.247+0000 7f650d683200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T18:59:58.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:58.357+0000 7f650d683200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T18:59:58.962 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:58.468+0000 7f650d683200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T18:59:58.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:58.675+0000 7f650d683200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T18:59:59.383 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:59.073+0000 7f650d683200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T18:59:59.384 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:59.181+0000 7f650d683200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T18:59:59.712 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:59.332+0000 7f650d683200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T18:59:59.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:59.447+0000 7f650d683200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T19:00:00.200 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 18:59:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T18:59:59.964+0000 7f650d683200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T19:00:01.038 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:00 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3828303378' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T19:00:01.039 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:00.670+0000 7f650d683200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T19:00:01.039 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:00.873+0000 7f650d683200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T19:00:01.097 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:01.098 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:01.098 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsid": "7541eb30-272f-11ee-9b34-001a4aab830c", 2023-07-20T19:00:01.098 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T19:00:01.098 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T19:00:01.098 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 0 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:01.099 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T19:00:01.100 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T19:00:01.101 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.102 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T19:00:01.102 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:01.102 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T19:00:01.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T19:00:01.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T19:00:01.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T19:00:01.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T19:00:01.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:01.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T19:00:01.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:59:47.497018+0000", 2023-07-20T19:00:01.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.534+0000 7fb4fb1ce700 1 Processor -- start 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.534+0000 7fb4fb1ce700 1 -- start start 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4fb1ce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4fb1ce700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4079eb0 con 0x7fb4f41069d0 2023-07-20T19:00:01.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4f8f6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4f8f6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57150/0 (socket says 172.21.15.61:57150) 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4f8f6a700 1 -- 172.21.15.61:0/1271614605 learned_addr learned my addr 172.21.15.61:0/1271614605 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4f8f6a700 1 -- 172.21.15.61:0/1271614605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4109300 con 0x7fb4f41069d0 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.535+0000 7fb4f8f6a700 1 --2- 172.21.15.61:0/1271614605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4108dc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80098a0 tx=0x7fb4e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cb2fcb34330155 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4f37fe700 1 -- 172.21.15.61:0/1271614605 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e8004160 con 0x7fb4f41069d0 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4f37fe700 1 -- 172.21.15.61:0/1271614605 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4e80042c0 con 0x7fb4f41069d0 2023-07-20T19:00:01.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4f37fe700 1 -- 172.21.15.61:0/1271614605 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e8011a00 con 0x7fb4f41069d0 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/1271614605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 msgr2=0x7fb4f4108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4fb1ce700 1 --2- 172.21.15.61:0/1271614605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4108dc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80098a0 tx=0x7fb4e8009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/1271614605 shutdown_connections 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4fb1ce700 1 --2- 172.21.15.61:0/1271614605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4108dc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/1271614605 >> 172.21.15.61:0/1271614605 conn(0x7fb4f4100400 msgr2=0x7fb4f4102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.536+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/1271614605 shutdown_connections 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.537+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/1271614605 wait complete. 2023-07-20T19:00:01.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.537+0000 7fb4fb1ce700 1 Processor -- start 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.537+0000 7fb4fb1ce700 1 -- start start 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4fb1ce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4fb1ce700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f419a170 con 0x7fb4f41069d0 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f8f6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f8f6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57152/0 (socket says 172.21.15.61:57152) 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f8f6a700 1 -- 172.21.15.61:0/3828303378 learned_addr learned my addr 172.21.15.61:0/3828303378 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f8f6a700 1 -- 172.21.15.61:0/3828303378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e8009550 con 0x7fb4f41069d0 2023-07-20T19:00:01.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f8f6a700 1 --2- 172.21.15.61:0/3828303378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4199c30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009fb0 tx=0x7fb4e8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e8003a40 con 0x7fb4f41069d0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f419a370 con 0x7fb4f41069d0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4e8003ba0 con 0x7fb4f41069d0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.538+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e801a750 con 0x7fb4f41069d0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.539+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f419a790 con 0x7fb4f41069d0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.539+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb4e801acd0 con 0x7fb4f41069d0 2023-07-20T19:00:01.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.539+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb4e80193c0 con 0x7fb4f41069d0 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.540+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f4061da0 con 0x7fb4f41069d0 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.544+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb4e8024b60 con 0x7fb4f41069d0 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.677+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb4f419d280 con 0x7fb4f41069d0 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.677+0000 7fb4f1ffb700 1 -- 172.21.15.61:0/3828303378 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb4e8024430 con 0x7fb4f41069d0 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.678+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 msgr2=0x7fb4f4199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.678+0000 7fb4fb1ce700 1 --2- 172.21.15.61:0/3828303378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4199c30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009fb0 tx=0x7fb4e8011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.679+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 shutdown_connections 2023-07-20T19:00:01.117 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.679+0000 7fb4fb1ce700 1 --2- 172.21.15.61:0/3828303378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4f41069d0 0x7fb4f4199c30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:01.118 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.679+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 >> 172.21.15.61:0/3828303378 conn(0x7fb4f4100400 msgr2=0x7fb4f4101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:01.118 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.679+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 shutdown_connections 2023-07-20T19:00:01.119 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:00.679+0000 7fb4fb1ce700 1 -- 172.21.15.61:0/3828303378 wait complete. 2023-07-20T19:00:01.119 INFO:teuthology.orchestra.run.smithi061.stdout:mgr not available, waiting (1/15)... 2023-07-20T19:00:02.212 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:01.848+0000 7f650d683200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:00:02.213 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:01.961+0000 7f650d683200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T19:00:02.579 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:02.179+0000 7f650d683200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:00:02.835 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:02.527+0000 7f650d683200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T19:00:03.212 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:02.784+0000 7f650d683200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T19:00:03.213 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:02.966+0000 7f650d683200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T19:00:03.712 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:03 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:03.309+0000 7f650d683200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T19:00:04.195 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:03 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2296330756' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T19:00:04.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:04.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:04.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsid": "7541eb30-272f-11ee-9b34-001a4aab830c", 2023-07-20T19:00:04.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T19:00:04.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T19:00:04.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T19:00:04.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T19:00:04.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T19:00:04.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T19:00:04.271 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 0 2023-07-20T19:00:04.271 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:04.271 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T19:00:04.272 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T19:00:04.272 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:04.272 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-07-20T19:00:04.272 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T19:00:04.273 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:04.273 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T19:00:04.273 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T19:00:04.274 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.274 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T19:00:04.274 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:04.274 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T19:00:04.274 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T19:00:04.275 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T19:00:04.275 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T19:00:04.275 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T19:00:04.275 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T19:00:04.276 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.276 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T19:00:04.276 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T19:00:04.276 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T19:00:04.277 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T19:00:04.277 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T19:00:04.277 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T19:00:04.277 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T19:00:04.278 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T19:00:04.279 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T19:00:04.280 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T19:00:04.280 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:04.280 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:04.281 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.281 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:59:47.497018+0000", 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:04.282 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6f4a8b700 1 Processor -- start 2023-07-20T19:00:04.283 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6f4a8b700 1 -- start start 2023-07-20T19:00:04.283 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6f4a8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:04.283 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6f4a8b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f0079eb0 con 0x7fd6f01069d0 2023-07-20T19:00:04.283 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6ee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:04.283 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6ee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57154/0 (socket says 172.21.15.61:57154) 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.535+0000 7fd6ee59c700 1 -- 172.21.15.61:0/3202503269 learned_addr learned my addr 172.21.15.61:0/3202503269 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.536+0000 7fd6ee59c700 1 -- 172.21.15.61:0/3202503269 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f0109300 con 0x7fd6f01069d0 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.536+0000 7fd6ee59c700 1 --2- 172.21.15.61:0/3202503269 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098a0 tx=0x7fd6e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2864bb4f38f83e91 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.536+0000 7fd6ed59a700 1 -- 172.21.15.61:0/3202503269 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0004160 con 0x7fd6f01069d0 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.536+0000 7fd6ed59a700 1 -- 172.21.15.61:0/3202503269 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6e00042c0 con 0x7fd6f01069d0 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.536+0000 7fd6ed59a700 1 -- 172.21.15.61:0/3202503269 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0011a00 con 0x7fd6f01069d0 2023-07-20T19:00:04.284 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/3202503269 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 msgr2=0x7fd6f0108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 --2- 172.21.15.61:0/3202503269 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098a0 tx=0x7fd6e0009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/3202503269 shutdown_connections 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 --2- 172.21.15.61:0/3202503269 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0108dc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/3202503269 >> 172.21.15.61:0/3202503269 conn(0x7fd6f0100400 msgr2=0x7fd6f0102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/3202503269 shutdown_connections 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.537+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/3202503269 wait complete. 2023-07-20T19:00:04.285 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6f4a8b700 1 Processor -- start 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6f4a8b700 1 -- start start 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6f4a8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6f4a8b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019a170 con 0x7fd6f01069d0 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6ee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6ee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57156/0 (socket says 172.21.15.61:57156) 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6ee59c700 1 -- 172.21.15.61:0/2296330756 learned_addr learned my addr 172.21.15.61:0/2296330756 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.538+0000 7fd6ee59c700 1 -- 172.21.15.61:0/2296330756 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e0009550 con 0x7fd6f01069d0 2023-07-20T19:00:04.286 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6ee59c700 1 --2- 172.21.15.61:0/2296330756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0199c30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009fb0 tx=0x7fd6e0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0003a40 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f019a370 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6e0003ba0 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6e001a750 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f019a790 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd6e001acd0 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.539+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd6e00193c0 con 0x7fd6f01069d0 2023-07-20T19:00:04.287 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.540+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f010a100 con 0x7fd6f01069d0 2023-07-20T19:00:04.288 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.542+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd6e0024b60 con 0x7fd6f01069d0 2023-07-20T19:00:04.288 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.674+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd6f010a2f0 con 0x7fd6f01069d0 2023-07-20T19:00:04.288 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.674+0000 7fd6df7fe700 1 -- 172.21.15.61:0/2296330756 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd6e0024d00 con 0x7fd6f01069d0 2023-07-20T19:00:04.288 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 msgr2=0x7fd6f0199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:04.288 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 --2- 172.21.15.61:0/2296330756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0199c30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009fb0 tx=0x7fd6e0011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:04.289 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 shutdown_connections 2023-07-20T19:00:04.289 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 --2- 172.21.15.61:0/2296330756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6f01069d0 0x7fd6f0199c30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:04.289 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 >> 172.21.15.61:0/2296330756 conn(0x7fd6f0100400 msgr2=0x7fd6f0101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:04.289 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 shutdown_connections 2023-07-20T19:00:04.289 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:03.675+0000 7fd6f4a8b700 1 -- 172.21.15.61:0/2296330756 wait complete. 2023-07-20T19:00:04.289 INFO:teuthology.orchestra.run.smithi061.stdout:mgr not available, waiting (2/15)... 2023-07-20T19:00:04.462 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:04.143+0000 7f650d683200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T19:00:04.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:04.302+0000 7f650d683200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T19:00:04.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:04.544+0000 7f650d683200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T19:00:04.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:04.681+0000 7f650d683200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T19:00:05.462 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:04.952+0000 7f650d683200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T19:00:06.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:05.993+0000 7f650d683200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T19:00:06.463 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: Activating manager daemon x 2023-07-20T19:00:06.463 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: mgrmap e2: x(active, starting, since 0.00330585s) 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: Manager daemon x is now available 2023-07-20T19:00:06.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T19:00:06.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T19:00:06.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' 2023-07-20T19:00:06.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:06 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' 2023-07-20T19:00:07.174 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:07 smithi061 ceph-mon[127454]: from='mgr.14100 172.21.15.61:0/2240576184' entity='mgr.x' 2023-07-20T19:00:07.174 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:07 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3777140951' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T19:00:07.174 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:07 smithi061 ceph-mon[127454]: mgrmap e3: x(active, since 1.0057s) 2023-07-20T19:00:07.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:07.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:07.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsid": "7541eb30-272f-11ee-9b34-001a4aab830c", 2023-07-20T19:00:07.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T19:00:07.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T19:00:07.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T19:00:07.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T19:00:07.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T19:00:07.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T19:00:07.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 0 2023-07-20T19:00:07.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:07.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T19:00:07.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T19:00:07.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T19:00:07.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:07.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T19:00:07.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T19:00:07.222 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T19:00:07.223 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T19:00:07.224 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T19:00:07.225 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:07.226 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:59:47.497018+0000", 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:07.227 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.662+0000 7f69a5ade700 1 Processor -- start 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.662+0000 7f69a5ade700 1 -- start start 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.662+0000 7f69a5ade700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0106390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.662+0000 7f69a5ade700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0106960 con 0x7f69a0105f70 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57172/0 (socket says 172.21.15.61:57172) 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699f7fe700 1 -- 172.21.15.61:0/1657219112 learned_addr learned my addr 172.21.15.61:0/1657219112 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:07.228 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699f7fe700 1 -- 172.21.15.61:0/1657219112 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a0107170 con 0x7f69a0105f70 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699f7fe700 1 --2- 172.21.15.61:0/1657219112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0106390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69940098a0 tx=0x7f6994009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbcb5cd9f98ccee server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699e7fc700 1 -- 172.21.15.61:0/1657219112 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6994004160 con 0x7f69a0105f70 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699e7fc700 1 -- 172.21.15.61:0/1657219112 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69940042c0 con 0x7f69a0105f70 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.663+0000 7f699e7fc700 1 -- 172.21.15.61:0/1657219112 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6994011a00 con 0x7f69a0105f70 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 -- 172.21.15.61:0/1657219112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 msgr2=0x7f69a0106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 --2- 172.21.15.61:0/1657219112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0106390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69940098a0 tx=0x7f6994009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:07.229 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 -- 172.21.15.61:0/1657219112 shutdown_connections 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 --2- 172.21.15.61:0/1657219112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0106390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 -- 172.21.15.61:0/1657219112 >> 172.21.15.61:0/1657219112 conn(0x7f69a0101550 msgr2=0x7f69a0103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 -- 172.21.15.61:0/1657219112 shutdown_connections 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.664+0000 7f69a5ade700 1 -- 172.21.15.61:0/1657219112 wait complete. 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f69a5ade700 1 Processor -- start 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f69a5ade700 1 -- start start 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f69a5ade700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:07.230 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f69a5ade700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a019a190 con 0x7f69a0105f70 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f699f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f699f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57174/0 (socket says 172.21.15.61:57174) 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.665+0000 7f699f7fe700 1 -- 172.21.15.61:0/3777140951 learned_addr learned my addr 172.21.15.61:0/3777140951 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f699f7fe700 1 -- 172.21.15.61:0/3777140951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994009550 con 0x7f69a0105f70 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f699f7fe700 1 --2- 172.21.15.61:0/3777140951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0199c50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6994006af0 tx=0x7f6994011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6994003a40 con 0x7f69a0105f70 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6994003ba0 con 0x7f69a0105f70 2023-07-20T19:00:07.231 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a019a390 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f699401a700 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.666+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a019a7b0 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.667+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 2) v1 ==== 45239+0+0 (secure 0 0 0) 0x7f699401a930 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.667+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0061da0 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.667+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f699404c200 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.669+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f699401abe0 con 0x7f69a0105f70 2023-07-20T19:00:07.232 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.801+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f69a010a360 con 0x7f69a0105f70 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.801+0000 7f699cff9700 1 -- 172.21.15.61:0/3777140951 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f6994018350 con 0x7f69a0105f70 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.802+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 msgr2=0x7f69a0199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.802+0000 7f69a5ade700 1 --2- 172.21.15.61:0/3777140951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0199c50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6994006af0 tx=0x7f6994011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.802+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 shutdown_connections 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.802+0000 7f69a5ade700 1 --2- 172.21.15.61:0/3777140951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69a0105f70 0x7f69a0199c50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.802+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 >> 172.21.15.61:0/3777140951 conn(0x7f69a0101550 msgr2=0x7f69a0102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:07.233 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.803+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 shutdown_connections 2023-07-20T19:00:07.234 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:06.803+0000 7f69a5ade700 1 -- 172.21.15.61:0/3777140951 wait complete. 2023-07-20T19:00:07.234 INFO:teuthology.orchestra.run.smithi061.stdout:mgr not available, waiting (3/15)... 2023-07-20T19:00:09.713 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:09 smithi061 ceph-mon[127454]: mgrmap e4: x(active, since 2s) 2023-07-20T19:00:10.663 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:10 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/226923906' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T19:00:10.668 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsid": "7541eb30-272f-11ee-9b34-001a4aab830c", 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.669 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 0 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:10.670 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2023-07-20T19:00:10.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T19:00:10.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:10.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T19:00:10.671 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T19:00:10.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T19:00:10.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:10.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T19:00:10.672 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T19:00:10.679 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T19:00:10.680 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T19:00:10.680 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T19:00:10.680 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T19:00:10.680 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.680 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T19:00:10.680 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T19:00:10.681 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T19:00:10.682 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ], 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:10.683 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.684 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T19:00:10.684 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T19:00:10.684 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T18:59:47.497018+0000", 2023-07-20T19:00:10.684 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T19:00:10.684 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout }, 2023-07-20T19:00:10.684 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.729+0000 7f08888c1700 1 Processor -- start 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.729+0000 7f08888c1700 1 -- start start 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.729+0000 7f08888c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884108db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.729+0000 7f08888c1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0884079eb0 con 0x7f08841069c0 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884108db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:10.685 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884108db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57186/0 (socket says 172.21.15.61:57186) 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088259c700 1 -- 172.21.15.61:0/252190709 learned_addr learned my addr 172.21.15.61:0/252190709 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088259c700 1 -- 172.21.15.61:0/252190709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08841092f0 con 0x7f08841069c0 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088259c700 1 --2- 172.21.15.61:0/252190709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884108db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f08780098a0 tx=0x7f0878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d98cf55205c968 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088159a700 1 -- 172.21.15.61:0/252190709 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0878004160 con 0x7f08841069c0 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088159a700 1 -- 172.21.15.61:0/252190709 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08780042c0 con 0x7f08841069c0 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.730+0000 7f088159a700 1 -- 172.21.15.61:0/252190709 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0878011a00 con 0x7f08841069c0 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 -- 172.21.15.61:0/252190709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 msgr2=0x7f0884108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:10.686 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 --2- 172.21.15.61:0/252190709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884108db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f08780098a0 tx=0x7f0878009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 -- 172.21.15.61:0/252190709 shutdown_connections 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 --2- 172.21.15.61:0/252190709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884108db0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 -- 172.21.15.61:0/252190709 >> 172.21.15.61:0/252190709 conn(0x7f08841003f0 msgr2=0x7f0884102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 -- 172.21.15.61:0/252190709 shutdown_connections 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.731+0000 7f08888c1700 1 -- 172.21.15.61:0/252190709 wait complete. 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f08888c1700 1 Processor -- start 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f08888c1700 1 -- start start 2023-07-20T19:00:10.687 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f08888c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884106320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:10.688 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f08888c1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0884106860 con 0x7f08841069c0 2023-07-20T19:00:10.688 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f088259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:10.688 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f088259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884106320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57188/0 (socket says 172.21.15.61:57188) 2023-07-20T19:00:10.688 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f088259c700 1 -- 172.21.15.61:0/226923906 learned_addr learned my addr 172.21.15.61:0/226923906 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:10.688 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.732+0000 7f088259c700 1 -- 172.21.15.61:0/226923906 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878009550 con 0x7f08841069c0 2023-07-20T19:00:10.689 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.733+0000 7f088259c700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884106320 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0878009fb0 tx=0x7f0878011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:10.689 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.733+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0878003a40 con 0x7f08841069c0 2023-07-20T19:00:10.689 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.733+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0878003ba0 con 0x7f08841069c0 2023-07-20T19:00:10.689 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.733+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0884104970 con 0x7f08841069c0 2023-07-20T19:00:10.690 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.733+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f087801a750 con 0x7f08841069c0 2023-07-20T19:00:10.690 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.733+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0884104d90 con 0x7f08841069c0 2023-07-20T19:00:10.690 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.734+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f087801a980 con 0x7f08841069c0 2023-07-20T19:00:10.691 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.734+0000 7f08737fe700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f086c0386c0 0x7f086c03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:10.691 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.734+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088410a0f0 con 0x7f08841069c0 2023-07-20T19:00:10.691 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.734+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f087804c4a0 con 0x7f08841069c0 2023-07-20T19:00:10.691 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.734+0000 7f0881d9b700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f086c0386c0 0x7f086c03ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:10.692 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.735+0000 7f0881d9b700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f086c0386c0 0x7f086c03ab80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0874006fd0 tx=0x7f0874006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:10.692 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:09.739+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f087801ac30 con 0x7f08841069c0 2023-07-20T19:00:10.692 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.254+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f088410a310 con 0x7f08841069c0 2023-07-20T19:00:10.692 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.254+0000 7f08737fe700 1 -- 172.21.15.61:0/226923906 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0878040180 con 0x7f08841069c0 2023-07-20T19:00:10.693 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f086c0386c0 msgr2=0x7f086c03ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:10.693 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f086c0386c0 0x7f086c03ab80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0874006fd0 tx=0x7f0874006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:10.693 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 msgr2=0x7f0884106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:10.694 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884106320 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0878009fb0 tx=0x7f0878011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:10.694 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 shutdown_connections 2023-07-20T19:00:10.694 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f086c0386c0 0x7f086c03ab80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:10.695 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 --2- 172.21.15.61:0/226923906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08841069c0 0x7f0884106320 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:10.695 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 >> 172.21.15.61:0/226923906 conn(0x7f08841003f0 msgr2=0x7f0884101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:10.695 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 shutdown_connections 2023-07-20T19:00:10.695 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:10.257+0000 7f08888c1700 1 -- 172.21.15.61:0/226923906 wait complete. 2023-07-20T19:00:10.696 INFO:teuthology.orchestra.run.smithi061.stdout:mgr is available 2023-07-20T19:00:12.073 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:11 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2569783728' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T19:00:12.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:12.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T19:00:12.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout fsid = 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:00:12.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T19:00:12.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T19:00:12.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T19:00:12.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T19:00:12.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T19:00:12.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.313+0000 7ff53bf69700 1 Processor -- start 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.313+0000 7ff53bf69700 1 -- start start 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.313+0000 7ff53bf69700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534106390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.313+0000 7ff53bf69700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534106960 con 0x7ff534105f70 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff539d05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:12.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff539d05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57192/0 (socket says 172.21.15.61:57192) 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff539d05700 1 -- 172.21.15.61:0/2134909799 learned_addr learned my addr 172.21.15.61:0/2134909799 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff539d05700 1 -- 172.21.15.61:0/2134909799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534107170 con 0x7ff534105f70 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff539d05700 1 --2- 172.21.15.61:0/2134909799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534106390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff5240098a0 tx=0x7ff524009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2d840b23bb8570 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff538d03700 1 -- 172.21.15.61:0/2134909799 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff524004160 con 0x7ff534105f70 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff538d03700 1 -- 172.21.15.61:0/2134909799 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5240042c0 con 0x7ff534105f70 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.314+0000 7ff538d03700 1 -- 172.21.15.61:0/2134909799 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff524011a00 con 0x7ff534105f70 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 -- 172.21.15.61:0/2134909799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 msgr2=0x7ff534106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:12.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 --2- 172.21.15.61:0/2134909799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534106390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff5240098a0 tx=0x7ff524009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:12.128 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 -- 172.21.15.61:0/2134909799 shutdown_connections 2023-07-20T19:00:12.128 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 --2- 172.21.15.61:0/2134909799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534106390 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:12.128 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 -- 172.21.15.61:0/2134909799 >> 172.21.15.61:0/2134909799 conn(0x7ff534101550 msgr2=0x7ff534103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:12.128 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 -- 172.21.15.61:0/2134909799 shutdown_connections 2023-07-20T19:00:12.128 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 -- 172.21.15.61:0/2134909799 wait complete. 2023-07-20T19:00:12.128 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.315+0000 7ff53bf69700 1 Processor -- start 2023-07-20T19:00:12.129 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff53bf69700 1 -- start start 2023-07-20T19:00:12.129 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff53bf69700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:12.130 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff53bf69700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534197f50 con 0x7ff534105f70 2023-07-20T19:00:12.130 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff539d05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:12.130 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff539d05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57194/0 (socket says 172.21.15.61:57194) 2023-07-20T19:00:12.130 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff539d05700 1 -- 172.21.15.61:0/2569783728 learned_addr learned my addr 172.21.15.61:0/2569783728 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:12.130 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff539d05700 1 -- 172.21.15.61:0/2569783728 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524009550 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff539d05700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534197a10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff524006af0 tx=0x7ff524011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff524003a40 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff524003ba0 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff52401a700 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff534198150 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.316+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff534198570 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.317+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7ff52401a930 con 0x7ff534105f70 2023-07-20T19:00:12.131 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.317+0000 7ff52affd700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7ff52003cb50 0x7ff52003f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.317+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff52404c600 con 0x7ff534105f70 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.318+0000 7ff539504700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7ff52003cb50 0x7ff52003f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.318+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff534191860 con 0x7ff534105f70 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.318+0000 7ff539504700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7ff52003cb50 0x7ff52003f010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff530006fd0 tx=0x7ff530006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.322+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff52401abe0 con 0x7ff534105f70 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.676+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff53407a0e0 con 0x7ff534105f70 2023-07-20T19:00:12.132 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.677+0000 7ff52affd700 1 -- 172.21.15.61:0/2569783728 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7ff524018350 con 0x7ff534105f70 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7ff52003cb50 msgr2=0x7ff52003f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7ff52003cb50 0x7ff52003f010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff530006fd0 tx=0x7ff530006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 msgr2=0x7ff534197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534197a10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff524006af0 tx=0x7ff524011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 shutdown_connections 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7ff52003cb50 0x7ff52003f010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:12.133 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 --2- 172.21.15.61:0/2569783728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff534105f70 0x7ff534197a10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:12.134 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 >> 172.21.15.61:0/2569783728 conn(0x7ff534101550 msgr2=0x7ff534102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:12.134 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 shutdown_connections 2023-07-20T19:00:12.134 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:11.680+0000 7ff53bf69700 1 -- 172.21.15.61:0/2569783728 wait complete. 2023-07-20T19:00:12.134 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling cephadm module... 2023-07-20T19:00:13.784 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3849767318' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T19:00:13.784 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: ignoring --setuser ceph since I am not root 2023-07-20T19:00:13.785 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: ignoring --setgroup ceph since I am not root 2023-07-20T19:00:13.785 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:13.634+0000 7f37e6c89200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T19:00:13.785 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:13.730+0000 7f37e6c89200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T19:00:13.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.536+0000 7fea0d212700 1 Processor -- start 2023-07-20T19:00:13.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.536+0000 7fea0d212700 1 -- start start 2023-07-20T19:00:13.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea0d212700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:13.861 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea0d212700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea08079e70 con 0x7fea08106a10 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea06d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea06d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57198/0 (socket says 172.21.15.61:57198) 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea06d9d700 1 -- 172.21.15.61:0/924848777 learned_addr learned my addr 172.21.15.61:0/924848777 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea06d9d700 1 -- 172.21.15.61:0/924848777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08109340 con 0x7fea08106a10 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea06d9d700 1 --2- 172.21.15.61:0/924848777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08108e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80098a0 tx=0x7fe9f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726d0ca5e100579a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea05d9b700 1 -- 172.21.15.61:0/924848777 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9f8004160 con 0x7fea08106a10 2023-07-20T19:00:13.862 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.537+0000 7fea05d9b700 1 -- 172.21.15.61:0/924848777 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9f80042c0 con 0x7fea08106a10 2023-07-20T19:00:13.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea05d9b700 1 -- 172.21.15.61:0/924848777 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9f8011a00 con 0x7fea08106a10 2023-07-20T19:00:13.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 -- 172.21.15.61:0/924848777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 msgr2=0x7fea08108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:13.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 --2- 172.21.15.61:0/924848777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08108e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80098a0 tx=0x7fe9f8009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:13.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 -- 172.21.15.61:0/924848777 shutdown_connections 2023-07-20T19:00:13.863 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 --2- 172.21.15.61:0/924848777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08108e00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:13.867 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 -- 172.21.15.61:0/924848777 >> 172.21.15.61:0/924848777 conn(0x7fea08100400 msgr2=0x7fea08102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:13.867 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 -- 172.21.15.61:0/924848777 shutdown_connections 2023-07-20T19:00:13.867 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.538+0000 7fea0d212700 1 -- 172.21.15.61:0/924848777 wait complete. 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea0d212700 1 Processor -- start 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea0d212700 1 -- start start 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea0d212700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea0d212700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0819a170 con 0x7fea08106a10 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea06d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea06d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57200/0 (socket says 172.21.15.61:57200) 2023-07-20T19:00:13.868 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.539+0000 7fea06d9d700 1 -- 172.21.15.61:0/3849767318 learned_addr learned my addr 172.21.15.61:0/3849767318 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fea06d9d700 1 -- 172.21.15.61:0/3849767318 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f8009550 con 0x7fea08106a10 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fea06d9d700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08199c30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009fb0 tx=0x7fe9f8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9f8003a40 con 0x7fea08106a10 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9f8003ba0 con 0x7fea08106a10 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0819a370 con 0x7fea08106a10 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9f801a750 con 0x7fea08106a10 2023-07-20T19:00:13.869 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.540+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0819a790 con 0x7fea08106a10 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.541+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fe9f801a980 con 0x7fea08106a10 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.541+0000 7fe9effff700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7fe9f00386c0 0x7fe9f003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.541+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0810a0e0 con 0x7fea08106a10 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.541+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe9f804c350 con 0x7fea08106a10 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.541+0000 7fea0659c700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7fe9f00386c0 0x7fe9f003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.542+0000 7fea0659c700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7fe9f00386c0 0x7fe9f003ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc006fd0 tx=0x7fe9fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:12.546+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe9f8018db0 con 0x7fea08106a10 2023-07-20T19:00:13.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.010+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fea0819d5b0 con 0x7fea08106a10 2023-07-20T19:00:13.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.445+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe9f8015db0 con 0x7fea08106a10 2023-07-20T19:00:13.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.446+0000 7fe9effff700 1 -- 172.21.15.61:0/3849767318 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe9f802c760 con 0x7fea08106a10 2023-07-20T19:00:13.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7fe9f00386c0 msgr2=0x7fe9f003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:13.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7fe9f00386c0 0x7fe9f003ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc006fd0 tx=0x7fe9fc006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:13.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 msgr2=0x7fea08199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:13.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08199c30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009fb0 tx=0x7fe9f8011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:13.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 shutdown_connections 2023-07-20T19:00:13.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7fe9f00386c0 0x7fe9f003ab80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:13.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 --2- 172.21.15.61:0/3849767318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea08106a10 0x7fea08199c30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:13.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 >> 172.21.15.61:0/3849767318 conn(0x7fea08100400 msgr2=0x7fea08101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:13.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 shutdown_connections 2023-07-20T19:00:13.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:13.449+0000 7fea0d212700 1 -- 172.21.15.61:0/3849767318 wait complete. 2023-07-20T19:00:14.079 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:13.843+0000 7f37e6c89200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T19:00:14.080 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:13.959+0000 7f37e6c89200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T19:00:14.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:14.186+0000 7f37e6c89200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T19:00:14.747 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:14.586+0000 7f37e6c89200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T19:00:14.747 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3849767318' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T19:00:14.748 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-mon[127454]: mgrmap e5: x(active, since 7s) 2023-07-20T19:00:15.018 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:14.695+0000 7f37e6c89200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T19:00:15.018 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:14.849+0000 7f37e6c89200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T19:00:15.018 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:14 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:14.966+0000 7f37e6c89200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T19:00:15.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:15.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T19:00:15.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T19:00:15.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T19:00:15.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T19:00:15.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:15.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f15852a6700 1 Processor -- start 2023-07-20T19:00:15.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f15852a6700 1 -- start start 2023-07-20T19:00:15.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f15852a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:15.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f15852a6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1580106cd0 con 0x7f1580106370 2023-07-20T19:00:15.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f157effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:15.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f157effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57214/0 (socket says 172.21.15.61:57214) 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.310+0000 7f157effd700 1 -- 172.21.15.61:0/2541791613 learned_addr learned my addr 172.21.15.61:0/2541791613 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.311+0000 7f157effd700 1 -- 172.21.15.61:0/2541791613 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580106e10 con 0x7f1580106370 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.311+0000 7f157effd700 1 --2- 172.21.15.61:0/2541791613 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580106790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15700098a0 tx=0x7f1570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e509423ff4ca74 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.311+0000 7f157dffb700 1 -- 172.21.15.61:0/2541791613 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1570004160 con 0x7f1580106370 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.311+0000 7f157dffb700 1 -- 172.21.15.61:0/2541791613 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15700042c0 con 0x7f1580106370 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.311+0000 7f157dffb700 1 -- 172.21.15.61:0/2541791613 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1570011a00 con 0x7f1580106370 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 -- 172.21.15.61:0/2541791613 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 msgr2=0x7f1580106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:15.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 --2- 172.21.15.61:0/2541791613 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580106790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15700098a0 tx=0x7f1570009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 -- 172.21.15.61:0/2541791613 shutdown_connections 2023-07-20T19:00:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 --2- 172.21.15.61:0/2541791613 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580106790 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 -- 172.21.15.61:0/2541791613 >> 172.21.15.61:0/2541791613 conn(0x7f15800ffbc0 msgr2=0x7f1580101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 -- 172.21.15.61:0/2541791613 shutdown_connections 2023-07-20T19:00:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 -- 172.21.15.61:0/2541791613 wait complete. 2023-07-20T19:00:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.312+0000 7f15852a6700 1 Processor -- start 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f15852a6700 1 -- start start 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f15852a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f15852a6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158019a1b0 con 0x7f1580106370 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f157effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f157effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57216/0 (socket says 172.21.15.61:57216) 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f157effd700 1 -- 172.21.15.61:0/3015627432 learned_addr learned my addr 172.21.15.61:0/3015627432 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:15.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f157effd700 1 -- 172.21.15.61:0/3015627432 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570009550 con 0x7f1580106370 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.313+0000 7f157effd700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580199c70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1570009fb0 tx=0x7f1570011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.314+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1570003a40 con 0x7f1580106370 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.314+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1570003ba0 con 0x7f1580106370 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.314+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158019a3b0 con 0x7f1580106370 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.314+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f157001a750 con 0x7f1580106370 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.314+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158019a7d0 con 0x7f1580106370 2023-07-20T19:00:15.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.315+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f157001a9d0 con 0x7f1580106370 2023-07-20T19:00:15.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.315+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580061da0 con 0x7f1580106370 2023-07-20T19:00:15.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.315+0000 7f1567fff700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 0x7f156803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:15.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.315+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f157004c390 con 0x7f1580106370 2023-07-20T19:00:15.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.315+0000 7f157e7fc700 1 -- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 msgr2=0x7f156803ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:15.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.315+0000 7f157e7fc700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 0x7f156803ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:00:15.217 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.320+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1570018b80 con 0x7f1580106370 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.516+0000 7f157e7fc700 1 -- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 msgr2=0x7f156803ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.516+0000 7f157e7fc700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 0x7f156803ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.783+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f158019d960 con 0x7f1580106370 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.784+0000 7f1567fff700 1 -- 172.21.15.61:0/3015627432 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1570018350 con 0x7f1580106370 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 msgr2=0x7f156803ab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 0x7f156803ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:15.218 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 msgr2=0x7f1580199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580199c70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1570009fb0 tx=0x7f1570011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 shutdown_connections 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f15680386c0 0x7f156803ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 --2- 172.21.15.61:0/3015627432 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1580106370 0x7f1580199c70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.786+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 >> 172.21.15.61:0/3015627432 conn(0x7f15800ffbc0 msgr2=0x7f15801012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.787+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 shutdown_connections 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:14.787+0000 7f15852a6700 1 -- 172.21.15.61:0/3015627432 wait complete. 2023-07-20T19:00:15.219 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for the mgr to restart... 2023-07-20T19:00:15.220 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for mgr epoch 5... 2023-07-20T19:00:15.713 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:15 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3015627432' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T19:00:15.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:15 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:15.493+0000 7f37e6c89200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T19:00:16.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:16 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:16.180+0000 7f37e6c89200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T19:00:16.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:16 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:16.372+0000 7f37e6c89200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T19:00:17.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:17 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:17.334+0000 7f37e6c89200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:00:17.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:17 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:17.445+0000 7f37e6c89200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T19:00:18.065 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:17 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:17.664+0000 7f37e6c89200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:00:18.320 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:18 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:18.013+0000 7f37e6c89200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T19:00:18.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:18 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:18.268+0000 7f37e6c89200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T19:00:18.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:18 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:18.448+0000 7f37e6c89200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T19:00:19.213 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:18 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:18.776+0000 7f37e6c89200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T19:00:19.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:19 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:19.600+0000 7f37e6c89200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T19:00:19.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:19 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:19.757+0000 7f37e6c89200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T19:00:20.459 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:19.999+0000 7f37e6c89200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T19:00:20.460 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:20.136+0000 7f37e6c89200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T19:00:20.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:20.406+0000 7f37e6c89200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T19:00:21.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:21.447+0000 7f37e6c89200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T19:00:21.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: Active manager daemon x restarted 2023-07-20T19:00:21.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: Activating manager daemon x 2023-07-20T19:00:21.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: mgrmap e6: x(active, starting, since 0.0035508s) 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T19:00:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: Manager daemon x is now available 2023-07-20T19:00:21.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:21.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:21.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:21.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:21 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:22.861 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: Found migration_current of "None". Setting to last migration. 2023-07-20T19:00:22.861 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:22.861 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T19:00:22.862 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T19:00:22.862 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:22.862 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:22.862 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T19:00:22.862 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: mgrmap e7: x(active, since 1.00638s) 2023-07-20T19:00:22.862 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:22 smithi061 ceph-mon[127454]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T19:00:22.870 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:22.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T19:00:22.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T19:00:22.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:22.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.650+0000 7f5ed77b1700 1 Processor -- start 2023-07-20T19:00:22.871 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.650+0000 7f5ed77b1700 1 -- start start 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.650+0000 7f5ed77b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.650+0000 7f5ed77b1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0079e70 con 0x7f5ed0106a20 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ed554d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ed554d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57224/0 (socket says 172.21.15.61:57224) 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ed554d700 1 -- 172.21.15.61:0/4861584 learned_addr learned my addr 172.21.15.61:0/4861584 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ed554d700 1 -- 172.21.15.61:0/4861584 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0109350 con 0x7f5ed0106a20 2023-07-20T19:00:22.872 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ed554d700 1 --2- 172.21.15.61:0/4861584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0108e10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0098a0 tx=0x7f5ecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6677dd6374f19a9d server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:22.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ec7fff700 1 -- 172.21.15.61:0/4861584 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ecc004160 con 0x7f5ed0106a20 2023-07-20T19:00:22.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ec7fff700 1 -- 172.21.15.61:0/4861584 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ecc0042c0 con 0x7f5ed0106a20 2023-07-20T19:00:22.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.651+0000 7f5ec7fff700 1 -- 172.21.15.61:0/4861584 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ecc011a00 con 0x7f5ed0106a20 2023-07-20T19:00:22.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4861584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 msgr2=0x7f5ed0108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:22.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4861584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0108e10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0098a0 tx=0x7f5ecc009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.873 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4861584 shutdown_connections 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4861584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0108e10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4861584 >> 172.21.15.61:0/4861584 conn(0x7f5ed0100430 msgr2=0x7f5ed0102850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4861584 shutdown_connections 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.652+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4861584 wait complete. 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.653+0000 7f5ed77b1700 1 Processor -- start 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.653+0000 7f5ed77b1700 1 -- start start 2023-07-20T19:00:22.874 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed77b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed77b1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed019a190 con 0x7f5ed0106a20 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed554d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed554d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57226/0 (socket says 172.21.15.61:57226) 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed554d700 1 -- 172.21.15.61:0/4271544474 learned_addr learned my addr 172.21.15.61:0/4271544474 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed554d700 1 -- 172.21.15.61:0/4271544474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc009550 con 0x7f5ed0106a20 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed554d700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0199c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009fb0 tx=0x7f5ecc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:22.875 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ecc003a40 con 0x7f5ed0106a20 2023-07-20T19:00:22.876 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed019a390 con 0x7f5ed0106a20 2023-07-20T19:00:22.876 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ecc003ba0 con 0x7f5ed0106a20 2023-07-20T19:00:22.876 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ecc01a750 con 0x7f5ed0106a20 2023-07-20T19:00:22.876 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.654+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed019a7b0 con 0x7f5ed0106a20 2023-07-20T19:00:22.876 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.655+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5ecc01a9d0 con 0x7f5ed0106a20 2023-07-20T19:00:22.876 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.656+0000 7f5ec67fc700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:22.877 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.656+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 --> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ed0061da0 con 0x7f5ebc0386c0 2023-07-20T19:00:22.877 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.656+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ecc024430 con 0x7f5ed0106a20 2023-07-20T19:00:22.877 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.656+0000 7f5ed4d4c700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 msgr2=0x7f5ebc03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:22.877 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.656+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:00:22.877 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.857+0000 7f5ed4d4c700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 msgr2=0x7f5ebc03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:15.857+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:16.258+0000 7f5ed4d4c700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 msgr2=0x7f5ebc03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:16.258+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:17.059+0000 7f5ed4d4c700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 msgr2=0x7f5ebc03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:17.059+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:18.659+0000 7f5ed4d4c700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 msgr2=0x7f5ebc03ab80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2016235088 2023-07-20T19:00:22.878 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:18.659+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:21.462+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f5ecc0246c0 con 0x7f5ed0106a20 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:21.462+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 msgr2=0x7f5ebc03ab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:21.462+0000 7f5ec67fc700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.464+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f5ecc02d960 con 0x7f5ed0106a20 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.464+0000 7f5ec67fc700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5ebc03be40 0x7f5ebc03e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.464+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ed0061da0 con 0x7f5ebc03be40 2023-07-20T19:00:22.879 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.464+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5ebc03be40 0x7f5ebc03e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.465+0000 7f5ed4d4c700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5ebc03be40 0x7f5ebc03e230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0003a10 tx=0x7f5ec0009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.466+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f5ed0061da0 con 0x7f5ebc03be40 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.476+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5ed010a0f0 con 0x7f5ebc03be40 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.476+0000 7f5ec67fc700 1 -- 172.21.15.61:0/4271544474 <== mgr.14120 v2:172.21.15.61:6800/2098378944 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5ed010a0f0 con 0x7f5ebc03be40 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5ebc03be40 msgr2=0x7f5ebc03e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5ebc03be40 0x7f5ebc03e230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0003a10 tx=0x7f5ec0009290 comp rx=0 tx=0).stop 2023-07-20T19:00:22.880 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 msgr2=0x7f5ed0199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0199c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009fb0 tx=0x7f5ecc011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 shutdown_connections 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5ebc03be40 0x7f5ebc03e230 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ed0106a20 0x7f5ed0199c50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 --2- 172.21.15.61:0/4271544474 >> [v2:172.21.15.61:6800/2016235088,v1:172.21.15.61:6801/2016235088] conn(0x7f5ebc0386c0 0x7f5ebc03ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 >> 172.21.15.61:0/4271544474 conn(0x7f5ed0100430 msgr2=0x7f5ed0101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:22.881 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 shutdown_connections 2023-07-20T19:00:22.882 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:22.477+0000 7f5ed77b1700 1 -- 172.21.15.61:0/4271544474 wait complete. 2023-07-20T19:00:22.882 INFO:teuthology.orchestra.run.smithi061.stdout:mgr epoch 5 is available 2023-07-20T19:00:22.882 INFO:teuthology.orchestra.run.smithi061.stdout:Setting orchestrator backend to cephadm... 2023-07-20T19:00:23.871 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:23 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:22] ENGINE Bus STARTING 2023-07-20T19:00:23.872 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:23 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:23] ENGINE Serving on http://172.21.15.61:8765 2023-07-20T19:00:23.872 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:23 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:23] ENGINE Serving on https://172.21.15.61:7150 2023-07-20T19:00:23.872 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:23 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:23] ENGINE Bus STARTED 2023-07-20T19:00:23.872 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:23 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:24.120 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6ef1ba7700 1 Processor -- start 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6ef1ba7700 1 -- start start 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6ef1ba7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6ef1ba7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec079e70 con 0x7f6eec106a10 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6eeb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6eeb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57278/0 (socket says 172.21.15.61:57278) 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.271+0000 7f6eeb7fe700 1 -- 172.21.15.61:0/1435546648 learned_addr learned my addr 172.21.15.61:0/1435546648 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:24.121 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.272+0000 7f6eeb7fe700 1 -- 172.21.15.61:0/1435546648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec109340 con 0x7f6eec106a10 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.272+0000 7f6eeb7fe700 1 --2- 172.21.15.61:0/1435546648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec108e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0098a0 tx=0x7f6edc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2fecf08a1a324f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.272+0000 7f6eea7fc700 1 -- 172.21.15.61:0/1435546648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6edc004160 con 0x7f6eec106a10 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.272+0000 7f6eea7fc700 1 -- 172.21.15.61:0/1435546648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6edc0042c0 con 0x7f6eec106a10 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.272+0000 7f6eea7fc700 1 -- 172.21.15.61:0/1435546648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6edc011a00 con 0x7f6eec106a10 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1435546648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 msgr2=0x7f6eec108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 --2- 172.21.15.61:0/1435546648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec108e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0098a0 tx=0x7f6edc009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:24.122 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1435546648 shutdown_connections 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 --2- 172.21.15.61:0/1435546648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec108e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1435546648 >> 172.21.15.61:0/1435546648 conn(0x7f6eec100400 msgr2=0x7f6eec102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1435546648 shutdown_connections 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1435546648 wait complete. 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.273+0000 7f6ef1ba7700 1 Processor -- start 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6ef1ba7700 1 -- start start 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6ef1ba7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:24.123 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6ef1ba7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec197f50 con 0x7f6eec106a10 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6eeb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6eeb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57280/0 (socket says 172.21.15.61:57280) 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6eeb7fe700 1 -- 172.21.15.61:0/1953276924 learned_addr learned my addr 172.21.15.61:0/1953276924 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6eeb7fe700 1 -- 172.21.15.61:0/1953276924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc009550 con 0x7f6eec106a10 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.274+0000 7f6eeb7fe700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec197a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6edc009fb0 tx=0x7f6edc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.275+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6edc003a40 con 0x7f6eec106a10 2023-07-20T19:00:24.124 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.275+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6edc003ba0 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.275+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6edc01a750 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.275+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec198150 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.275+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec198570 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.275+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f6edc01a980 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.276+0000 7f6ee8ff9700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f6ed403c9f0 0x7f6ed403eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.276+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6edc04c550 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.276+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec191e20 con 0x7f6eec106a10 2023-07-20T19:00:24.125 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.276+0000 7f6eeaffd700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f6ed403c9f0 0x7f6ed403eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.276+0000 7f6eeaffd700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f6ed403c9f0 0x7f6ed403eeb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0006fd0 tx=0x7f6ee0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.281+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6edc01ac30 con 0x7f6eec106a10 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.518+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6edc02d910 con 0x7f6eec106a10 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.668+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eec060bb0 con 0x7f6ed403c9f0 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.674+0000 7f6ee8ff9700 1 -- 172.21.15.61:0/1953276924 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6eec060bb0 con 0x7f6ed403c9f0 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.676+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f6ed403c9f0 msgr2=0x7f6ed403eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:24.126 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.676+0000 7f6ef1ba7700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f6ed403c9f0 0x7f6ed403eeb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0006fd0 tx=0x7f6ee0006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.676+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 msgr2=0x7f6eec197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.676+0000 7f6ef1ba7700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec197a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6edc009fb0 tx=0x7f6edc011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.677+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 shutdown_connections 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.677+0000 7f6ef1ba7700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f6ed403c9f0 0x7f6ed403eeb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.677+0000 7f6ef1ba7700 1 --2- 172.21.15.61:0/1953276924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6eec106a10 0x7f6eec197a10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.677+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 >> 172.21.15.61:0/1953276924 conn(0x7f6eec100400 msgr2=0x7f6eec1010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.677+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 shutdown_connections 2023-07-20T19:00:24.127 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:23.677+0000 7f6ef1ba7700 1 -- 172.21.15.61:0/1953276924 wait complete. 2023-07-20T19:00:24.950 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:24 smithi061 ceph-mon[127454]: mgrmap e8: x(active, since 2s) 2023-07-20T19:00:24.950 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:24 smithi061 ceph-mon[127454]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:24.950 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:24 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:24.951 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:24 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:25.262 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T19:00:25.262 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f14756cd700 1 Processor -- start 2023-07-20T19:00:25.262 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f14756cd700 1 -- start start 2023-07-20T19:00:25.262 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f14756cd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470108df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:25.262 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f14756cd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470079e70 con 0x7f1470106a00 2023-07-20T19:00:25.263 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f146effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470108df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:25.263 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f146effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470108df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57284/0 (socket says 172.21.15.61:57284) 2023-07-20T19:00:25.263 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f146effd700 1 -- 172.21.15.61:0/1390000184 learned_addr learned my addr 172.21.15.61:0/1390000184 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:25.263 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.498+0000 7f146effd700 1 -- 172.21.15.61:0/1390000184 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470109330 con 0x7f1470106a00 2023-07-20T19:00:25.263 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f146effd700 1 --2- 172.21.15.61:0/1390000184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470108df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f14600098a0 tx=0x7f1460009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400d443ecdc2fddd server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:25.263 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f146dffb700 1 -- 172.21.15.61:0/1390000184 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1460004160 con 0x7f1470106a00 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f146dffb700 1 -- 172.21.15.61:0/1390000184 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14600042c0 con 0x7f1470106a00 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f146dffb700 1 -- 172.21.15.61:0/1390000184 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1460011a00 con 0x7f1470106a00 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f14756cd700 1 -- 172.21.15.61:0/1390000184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 msgr2=0x7f1470108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f14756cd700 1 --2- 172.21.15.61:0/1390000184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470108df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f14600098a0 tx=0x7f1460009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f14756cd700 1 -- 172.21.15.61:0/1390000184 shutdown_connections 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f14756cd700 1 --2- 172.21.15.61:0/1390000184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470108df0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:25.264 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.499+0000 7f14756cd700 1 -- 172.21.15.61:0/1390000184 >> 172.21.15.61:0/1390000184 conn(0x7f14701003f0 msgr2=0x7f1470102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:25.265 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.500+0000 7f14756cd700 1 -- 172.21.15.61:0/1390000184 shutdown_connections 2023-07-20T19:00:25.265 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.500+0000 7f14756cd700 1 -- 172.21.15.61:0/1390000184 wait complete. 2023-07-20T19:00:25.265 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.500+0000 7f14756cd700 1 Processor -- start 2023-07-20T19:00:25.265 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.500+0000 7f14756cd700 1 -- start start 2023-07-20T19:00:25.265 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f14756cd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470106360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f14756cd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701068a0 con 0x7f1470106a00 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f146effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470106360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f146effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470106360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57286/0 (socket says 172.21.15.61:57286) 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f146effd700 1 -- 172.21.15.61:0/749135804 learned_addr learned my addr 172.21.15.61:0/749135804 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f146effd700 1 -- 172.21.15.61:0/749135804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1460009550 con 0x7f1470106a00 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f146effd700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470106360 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1460009fb0 tx=0x7f1460011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:25.266 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1460003a40 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1460003ba0 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f146001a750 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701049b0 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.501+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1470104dd0 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.502+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f146001a9d0 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.502+0000 7f1457fff700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f14580386c0 0x7f145803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.502+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f146004c7f0 con 0x7f1470106a00 2023-07-20T19:00:25.267 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.503+0000 7f146e7fc700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f14580386c0 0x7f145803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:25.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.503+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147010a0d0 con 0x7f1470106a00 2023-07-20T19:00:25.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.503+0000 7f146e7fc700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f14580386c0 0x7f145803ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1464006fd0 tx=0x7f1464006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:25.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.508+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f146001ae60 con 0x7f1470106a00 2023-07-20T19:00:25.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.869+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f14701026d0 con 0x7f14580386c0 2023-07-20T19:00:25.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.869+0000 7f1457fff700 1 -- 172.21.15.61:0/749135804 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f14701026d0 con 0x7f14580386c0 2023-07-20T19:00:25.268 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f14580386c0 msgr2=0x7f145803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:25.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f14580386c0 0x7f145803ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1464006fd0 tx=0x7f1464006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:25.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 msgr2=0x7f1470106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:25.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470106360 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1460009fb0 tx=0x7f1460011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:25.269 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 shutdown_connections 2023-07-20T19:00:25.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f14580386c0 0x7f145803ab80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:25.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 --2- 172.21.15.61:0/749135804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1470106a00 0x7f1470106360 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:25.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 >> 172.21.15.61:0/749135804 conn(0x7f14701003f0 msgr2=0x7f1470101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:25.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 shutdown_connections 2023-07-20T19:00:25.270 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:24.872+0000 7f14756cd700 1 -- 172.21.15.61:0/749135804 wait complete. 2023-07-20T19:00:25.270 INFO:teuthology.orchestra.run.smithi061.stdout:Generating ssh key... 2023-07-20T19:00:25.963 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:25 smithi061 ceph-mon[127454]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:26.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: Generating public/private rsa key pair. 2023-07-20T19:00:26.464 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: Your identification has been saved in /tmp/tmpn8fvm55_/key. 2023-07-20T19:00:26.464 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: Your public key has been saved in /tmp/tmpn8fvm55_/key.pub. 2023-07-20T19:00:26.464 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: The key fingerprint is: 2023-07-20T19:00:26.464 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: SHA256:t8rRzvhm3EHJZf4oDwUbJYIEt/uf6jKu6n35G1I4Wzg ceph-7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:00:26.465 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: The key's randomart image is: 2023-07-20T19:00:26.465 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: +---[RSA 3072]----+ 2023-07-20T19:00:26.465 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | .oo.. ... | 2023-07-20T19:00:26.466 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | ... .o.o | 2023-07-20T19:00:26.466 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | . . O | 2023-07-20T19:00:26.466 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | + = o | 2023-07-20T19:00:26.466 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | E +. . o | 2023-07-20T19:00:26.466 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | X .+ . .| 2023-07-20T19:00:26.467 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | +o=. = | 2023-07-20T19:00:26.467 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | . .=B+o... | 2023-07-20T19:00:26.467 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: | .o.o+=OOoo | 2023-07-20T19:00:26.467 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: +----[SHA256]-----+ 2023-07-20T19:00:26.504 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.665+0000 7f27e0aba700 1 Processor -- start 2023-07-20T19:00:26.508 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.665+0000 7f27e0aba700 1 -- start start 2023-07-20T19:00:26.508 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.665+0000 7f27e0aba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc106390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:26.508 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.665+0000 7f27e0aba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc106960 con 0x7f27dc105f70 2023-07-20T19:00:26.508 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27da59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27da59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57290/0 (socket says 172.21.15.61:57290) 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27da59c700 1 -- 172.21.15.61:0/3575888944 learned_addr learned my addr 172.21.15.61:0/3575888944 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27da59c700 1 -- 172.21.15.61:0/3575888944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc107170 con 0x7f27dc105f70 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27da59c700 1 --2- 172.21.15.61:0/3575888944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc106390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27d00098a0 tx=0x7f27d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5b2f5d4f99e080 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27d959a700 1 -- 172.21.15.61:0/3575888944 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27d0004160 con 0x7f27dc105f70 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27d959a700 1 -- 172.21.15.61:0/3575888944 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27d00042c0 con 0x7f27dc105f70 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.666+0000 7f27d959a700 1 -- 172.21.15.61:0/3575888944 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27d0011a00 con 0x7f27dc105f70 2023-07-20T19:00:26.509 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 -- 172.21.15.61:0/3575888944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 msgr2=0x7f27dc106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 --2- 172.21.15.61:0/3575888944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc106390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27d00098a0 tx=0x7f27d0009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 -- 172.21.15.61:0/3575888944 shutdown_connections 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 --2- 172.21.15.61:0/3575888944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc106390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 -- 172.21.15.61:0/3575888944 >> 172.21.15.61:0/3575888944 conn(0x7f27dc101550 msgr2=0x7f27dc103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 -- 172.21.15.61:0/3575888944 shutdown_connections 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.667+0000 7f27e0aba700 1 -- 172.21.15.61:0/3575888944 wait complete. 2023-07-20T19:00:26.510 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27e0aba700 1 Processor -- start 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27e0aba700 1 -- start start 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27e0aba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27e0aba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc07c190 con 0x7f27dc105f70 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27da59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27da59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc07bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57292/0 (socket says 172.21.15.61:57292) 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.668+0000 7f27da59c700 1 -- 172.21.15.61:0/806465739 learned_addr learned my addr 172.21.15.61:0/806465739 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27da59c700 1 -- 172.21.15.61:0/806465739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d0009550 con 0x7f27dc105f70 2023-07-20T19:00:26.511 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27da59c700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc07bc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f27d0006af0 tx=0x7f27d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27d0003a40 con 0x7f27dc105f70 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27d0003ba0 con 0x7f27dc105f70 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27e0aba700 1 -- 172.21.15.61:0/806465739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc07a2a0 con 0x7f27dc105f70 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27d001a700 con 0x7f27dc105f70 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.669+0000 7f27e0aba700 1 -- 172.21.15.61:0/806465739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc07a6c0 con 0x7f27dc105f70 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.670+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f27d001a980 con 0x7f27dc105f70 2023-07-20T19:00:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.670+0000 7f27cb7fe700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f27c4038710 0x7f27c403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.670+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f27d004c830 con 0x7f27dc105f70 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.670+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c00052f0 con 0x7f27dc105f70 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.670+0000 7f27d9d9b700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f27c4038710 0x7f27c403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.671+0000 7f27d9d9b700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f27c4038710 0x7f27c403abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f27cc006fd0 tx=0x7f27cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:25.675+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27d0023390 con 0x7f27dc105f70 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.031+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f27c0000bc0 con 0x7f27c4038710 2023-07-20T19:00:26.513 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.130+0000 7f27cb7fe700 1 -- 172.21.15.61:0/806465739 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f27c0000bc0 con 0x7f27c4038710 2023-07-20T19:00:26.514 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f27c4038710 msgr2=0x7f27c403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:26.514 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f27c4038710 0x7f27c403abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f27cc006fd0 tx=0x7f27cc006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:26.514 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 msgr2=0x7f27dc07bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:26.514 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc07bc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f27d0006af0 tx=0x7f27d0011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:26.515 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 shutdown_connections 2023-07-20T19:00:26.515 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f27c4038710 0x7f27c403abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:26.515 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 --2- 172.21.15.61:0/806465739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27dc105f70 0x7f27dc07bc50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:26.515 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.132+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 >> 172.21.15.61:0/806465739 conn(0x7f27dc101550 msgr2=0x7f27dc102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:26.516 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.133+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 shutdown_connections 2023-07-20T19:00:26.516 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.133+0000 7f27c97fa700 1 -- 172.21.15.61:0/806465739 wait complete. 2023-07-20T19:00:27.463 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:27 smithi061 ceph-mon[127454]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:27.463 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:27 smithi061 ceph-mon[127454]: Generating ssh key... 2023-07-20T19:00:27.463 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:27 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:27.464 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:27 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:27.757 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:00:27.758 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.960+0000 7f025eaee700 1 Processor -- start 2023-07-20T19:00:27.758 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.960+0000 7f025eaee700 1 -- start start 2023-07-20T19:00:27.758 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.960+0000 7f025eaee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f0258105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:27.758 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.960+0000 7f025eaee700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02581057b0 con 0x7f0258104e50 2023-07-20T19:00:27.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f025c88a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f0258105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:27.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f025c88a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f0258105270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57296/0 (socket says 172.21.15.61:57296) 2023-07-20T19:00:27.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f025c88a700 1 -- 172.21.15.61:0/2490205244 learned_addr learned my addr 172.21.15.61:0/2490205244 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:27.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f025c88a700 1 -- 172.21.15.61:0/2490205244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02581058f0 con 0x7f0258104e50 2023-07-20T19:00:27.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f025c88a700 1 --2- 172.21.15.61:0/2490205244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f0258105270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f024c0098a0 tx=0x7f024c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbe92c4de31fe94 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:27.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f02577fe700 1 -- 172.21.15.61:0/2490205244 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f024c004160 con 0x7f0258104e50 2023-07-20T19:00:27.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f02577fe700 1 -- 172.21.15.61:0/2490205244 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f024c0042c0 con 0x7f0258104e50 2023-07-20T19:00:27.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.961+0000 7f02577fe700 1 -- 172.21.15.61:0/2490205244 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f024c011a00 con 0x7f0258104e50 2023-07-20T19:00:27.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 -- 172.21.15.61:0/2490205244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 msgr2=0x7f0258105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:27.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 --2- 172.21.15.61:0/2490205244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f0258105270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f024c0098a0 tx=0x7f024c009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:27.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 -- 172.21.15.61:0/2490205244 shutdown_connections 2023-07-20T19:00:27.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 --2- 172.21.15.61:0/2490205244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f0258105270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:27.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 -- 172.21.15.61:0/2490205244 >> 172.21.15.61:0/2490205244 conn(0x7f02581003f0 msgr2=0x7f0258102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:27.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 -- 172.21.15.61:0/2490205244 shutdown_connections 2023-07-20T19:00:27.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.962+0000 7f025eaee700 1 -- 172.21.15.61:0/2490205244 wait complete. 2023-07-20T19:00:27.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025eaee700 1 Processor -- start 2023-07-20T19:00:27.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025eaee700 1 -- start start 2023-07-20T19:00:27.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025eaee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f025818ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:27.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025eaee700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0258190490 con 0x7f0258104e50 2023-07-20T19:00:27.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025c88a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f025818ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:27.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025c88a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f025818ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57298/0 (socket says 172.21.15.61:57298) 2023-07-20T19:00:27.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025c88a700 1 -- 172.21.15.61:0/2557114871 learned_addr learned my addr 172.21.15.61:0/2557114871 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:27.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025c88a700 1 -- 172.21.15.61:0/2557114871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c009550 con 0x7f0258104e50 2023-07-20T19:00:27.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f025c88a700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f025818ff50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f024c009130 tx=0x7f024c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:27.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.963+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f024c003b70 con 0x7f0258104e50 2023-07-20T19:00:27.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.964+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f024c003cd0 con 0x7f0258104e50 2023-07-20T19:00:27.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.964+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f024c0197d0 con 0x7f0258104e50 2023-07-20T19:00:27.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.964+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025818e5a0 con 0x7f0258104e50 2023-07-20T19:00:27.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.964+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025818e9c0 con 0x7f0258104e50 2023-07-20T19:00:27.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.965+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f024c01abb0 con 0x7f0258104e50 2023-07-20T19:00:27.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.965+0000 7f0255ffb700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f0248038710 0x7f024803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:27.767 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.965+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0258061da0 con 0x7f0258104e50 2023-07-20T19:00:27.767 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.965+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f024c04c920 con 0x7f0258104e50 2023-07-20T19:00:27.767 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.965+0000 7f0257fff700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f0248038710 0x7f024803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:27.767 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.965+0000 7f0257fff700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f0248038710 0x7f024803abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0244006fd0 tx=0x7f0244006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:27.768 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:26.970+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f024c01a430 con 0x7f0258104e50 2023-07-20T19:00:27.768 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.333+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f02581025b0 con 0x7f0248038710 2023-07-20T19:00:27.768 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.334+0000 7f0255ffb700 1 -- 172.21.15.61:0/2557114871 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f02581025b0 con 0x7f0248038710 2023-07-20T19:00:27.768 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.336+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f0248038710 msgr2=0x7f024803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:27.769 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.336+0000 7f025eaee700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f0248038710 0x7f024803abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0244006fd0 tx=0x7f0244006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:27.769 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.336+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 msgr2=0x7f025818ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:27.769 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.336+0000 7f025eaee700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f025818ff50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f024c009130 tx=0x7f024c011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:27.769 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.337+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 shutdown_connections 2023-07-20T19:00:27.770 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.337+0000 7f025eaee700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f0248038710 0x7f024803abd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:27.770 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.337+0000 7f025eaee700 1 --2- 172.21.15.61:0/2557114871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0258104e50 0x7f025818ff50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:27.770 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.337+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 >> 172.21.15.61:0/2557114871 conn(0x7f02581003f0 msgr2=0x7f0258079920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:27.770 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.337+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 shutdown_connections 2023-07-20T19:00:27.771 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:27.337+0000 7f025eaee700 1 -- 172.21.15.61:0/2557114871 wait complete. 2023-07-20T19:00:27.771 INFO:teuthology.orchestra.run.smithi061.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T19:00:27.771 INFO:teuthology.orchestra.run.smithi061.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T19:00:27.771 INFO:teuthology.orchestra.run.smithi061.stdout:Adding host smithi061... 2023-07-20T19:00:28.873 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:28 smithi061 ceph-mon[127454]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:29.713 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:29 smithi061 ceph-mon[127454]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi061", "addr": "172.21.15.61", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:29.754 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:29 smithi061 ceph-mon[127454]: Deploying cephadm binary to smithi061 2023-07-20T19:00:31.907 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout Added host 'smithi061' with addr '172.21.15.61' 2023-07-20T19:00:31.907 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.159+0000 7f5fa5a92700 1 Processor -- start 2023-07-20T19:00:31.907 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.159+0000 7f5fa5a92700 1 -- start start 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5fa5a92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0108e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5fa5a92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0079e30 con 0x7f5fa0106a50 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5f9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0108e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5f9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0108e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57302/0 (socket says 172.21.15.61:57302) 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5f9f7fe700 1 -- 172.21.15.61:0/1140584688 learned_addr learned my addr 172.21.15.61:0/1140584688 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5f9f7fe700 1 -- 172.21.15.61:0/1140584688 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0109380 con 0x7f5fa0106a50 2023-07-20T19:00:31.908 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5f9f7fe700 1 --2- 172.21.15.61:0/1140584688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0108e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5f900098a0 tx=0x7f5f90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc9f33f93dc8e4f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.160+0000 7f5f9e7fc700 1 -- 172.21.15.61:0/1140584688 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f90004160 con 0x7f5fa0106a50 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5f9e7fc700 1 -- 172.21.15.61:0/1140584688 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f900042c0 con 0x7f5fa0106a50 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5f9e7fc700 1 -- 172.21.15.61:0/1140584688 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f90011a00 con 0x7f5fa0106a50 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 -- 172.21.15.61:0/1140584688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 msgr2=0x7f5fa0108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 --2- 172.21.15.61:0/1140584688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0108e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5f900098a0 tx=0x7f5f90009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 -- 172.21.15.61:0/1140584688 shutdown_connections 2023-07-20T19:00:31.909 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 --2- 172.21.15.61:0/1140584688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0108e40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 -- 172.21.15.61:0/1140584688 >> 172.21.15.61:0/1140584688 conn(0x7f5fa0100400 msgr2=0x7f5fa0102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 -- 172.21.15.61:0/1140584688 shutdown_connections 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.161+0000 7f5fa5a92700 1 -- 172.21.15.61:0/1140584688 wait complete. 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5fa5a92700 1 Processor -- start 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5fa5a92700 1 -- start start 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5fa5a92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:31.910 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5fa5a92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa019a1b0 con 0x7f5fa0106a50 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5f9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5f9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57304/0 (socket says 172.21.15.61:57304) 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.162+0000 7f5f9f7fe700 1 -- 172.21.15.61:0/4104606040 learned_addr learned my addr 172.21.15.61:0/4104606040 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5f9f7fe700 1 -- 172.21.15.61:0/4104606040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90009550 con 0x7f5fa0106a50 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5f9f7fe700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0199c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009fb0 tx=0x7f5f90011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f90003a40 con 0x7f5fa0106a50 2023-07-20T19:00:31.911 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f90003ba0 con 0x7f5fa0106a50 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa019a3b0 con 0x7f5fa0106a50 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f9001a750 con 0x7f5fa0106a50 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.163+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa019a7d0 con 0x7f5fa0106a50 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.164+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa010a0d0 con 0x7f5fa0106a50 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.165+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5f90024b60 con 0x7f5fa0106a50 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.165+0000 7f5f9cff9700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5f88038710 0x7f5f8803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:31.912 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.165+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5f9004cb00 con 0x7f5fa0106a50 2023-07-20T19:00:31.913 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.165+0000 7f5f9effd700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5f88038710 0x7f5f8803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:31.913 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.165+0000 7f5f9effd700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5f88038710 0x7f5f8803abd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f94006fd0 tx=0x7f5f94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:31.913 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.169+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f9001a980 con 0x7f5fa0106a50 2023-07-20T19:00:31.913 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:28.533+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi061", "addr": "172.21.15.61", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fa0102730 con 0x7f5f88038710 2023-07-20T19:00:31.913 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.346+0000 7f5f9cff9700 1 -- 172.21.15.61:0/4104606040 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f5fa0102730 con 0x7f5f88038710 2023-07-20T19:00:31.913 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5f88038710 msgr2=0x7f5f8803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5f88038710 0x7f5f8803abd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f94006fd0 tx=0x7f5f94006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 msgr2=0x7f5fa0199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0199c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009fb0 tx=0x7f5f90011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 shutdown_connections 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f5f88038710 0x7f5f8803abd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 --2- 172.21.15.61:0/4104606040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fa0106a50 0x7f5fa0199c70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 >> 172.21.15.61:0/4104606040 conn(0x7f5fa0100400 msgr2=0x7f5fa0101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:31.914 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 shutdown_connections 2023-07-20T19:00:31.915 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:31.349+0000 7f5fa5a92700 1 -- 172.21.15.61:0/4104606040 wait complete. 2023-07-20T19:00:31.915 INFO:teuthology.orchestra.run.smithi061.stdout:Deploying unmanaged mon service... 2023-07-20T19:00:32.714 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:32 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:32.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:32 smithi061 ceph-mon[127454]: Added host smithi061 2023-07-20T19:00:32.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:32 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:34.758 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T19:00:34.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77c4e6700 1 Processor -- start 2023-07-20T19:00:34.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77c4e6700 1 -- start start 2023-07-20T19:00:34.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77c4e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774104670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:34.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77c4e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff774104c40 con 0x7ff774104250 2023-07-20T19:00:34.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77a282700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:34.759 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77a282700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57338/0 (socket says 172.21.15.61:57338) 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.835+0000 7ff77a282700 1 -- 172.21.15.61:0/3788585673 learned_addr learned my addr 172.21.15.61:0/3788585673 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff77a282700 1 -- 172.21.15.61:0/3788585673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7741071e0 con 0x7ff774104250 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff77a282700 1 --2- 172.21.15.61:0/3788585673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774104670 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7700098a0 tx=0x7ff770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9dbbf052c5150f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff779280700 1 -- 172.21.15.61:0/3788585673 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff770004160 con 0x7ff774104250 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff779280700 1 -- 172.21.15.61:0/3788585673 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7700042c0 con 0x7ff774104250 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff779280700 1 -- 172.21.15.61:0/3788585673 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff770011a00 con 0x7ff774104250 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff77c4e6700 1 -- 172.21.15.61:0/3788585673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 msgr2=0x7ff774104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.836+0000 7ff77c4e6700 1 --2- 172.21.15.61:0/3788585673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774104670 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7700098a0 tx=0x7ff770009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 -- 172.21.15.61:0/3788585673 shutdown_connections 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 --2- 172.21.15.61:0/3788585673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774104670 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 -- 172.21.15.61:0/3788585673 >> 172.21.15.61:0/3788585673 conn(0x7ff7740ffbc0 msgr2=0x7ff774101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 -- 172.21.15.61:0/3788585673 shutdown_connections 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 -- 172.21.15.61:0/3788585673 wait complete. 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 Processor -- start 2023-07-20T19:00:34.761 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.837+0000 7ff77c4e6700 1 -- start start 2023-07-20T19:00:34.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77c4e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:34.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77c4e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77419a1b0 con 0x7ff774104250 2023-07-20T19:00:34.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77a282700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:34.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77a282700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57340/0 (socket says 172.21.15.61:57340) 2023-07-20T19:00:34.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77a282700 1 -- 172.21.15.61:0/189473578 learned_addr learned my addr 172.21.15.61:0/189473578 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:34.762 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77a282700 1 -- 172.21.15.61:0/189473578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff770009550 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77a282700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774199c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff770009fb0 tx=0x7ff770011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff770003a40 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff770003ba0 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77419a3b0 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff77001a750 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.838+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77419a7d0 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.839+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff770024430 con 0x7ff774104250 2023-07-20T19:00:34.763 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.839+0000 7ff76b7fe700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff760038710 0x7ff76003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.840+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff77004c790 con 0x7ff774104250 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.840+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff774061da0 con 0x7ff774104250 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.840+0000 7ff779a81700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff760038710 0x7ff76003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.840+0000 7ff779a81700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff760038710 0x7ff76003abd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff764006fd0 tx=0x7ff764006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:33.845+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7700246e0 con 0x7ff774104250 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.244+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff774101ea0 con 0x7ff760038710 2023-07-20T19:00:34.764 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.249+0000 7ff76b7fe700 1 -- 172.21.15.61:0/189473578 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff774101ea0 con 0x7ff760038710 2023-07-20T19:00:34.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff760038710 msgr2=0x7ff76003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:34.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff760038710 0x7ff76003abd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff764006fd0 tx=0x7ff764006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:34.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 msgr2=0x7ff774199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:34.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774199c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff770009fb0 tx=0x7ff770011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:34.765 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 shutdown_connections 2023-07-20T19:00:34.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff760038710 0x7ff76003abd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:34.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 --2- 172.21.15.61:0/189473578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff774104250 0x7ff774199c70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:34.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.251+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 >> 172.21.15.61:0/189473578 conn(0x7ff7740ffbc0 msgr2=0x7ff774101240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:34.766 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.252+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 shutdown_connections 2023-07-20T19:00:34.767 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:34.252+0000 7ff77c4e6700 1 -- 172.21.15.61:0/189473578 wait complete. 2023-07-20T19:00:34.767 INFO:teuthology.orchestra.run.smithi061.stdout:Deploying unmanaged mgr service... 2023-07-20T19:00:35.713 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:35 smithi061 ceph-mon[127454]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:35.713 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:35 smithi061 ceph-mon[127454]: Saving service mon spec with placement count:5 2023-07-20T19:00:35.714 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:35 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:36.102 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T19:00:36.102 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22d1a18700 1 Processor -- start 2023-07-20T19:00:36.102 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22d1a18700 1 -- start start 2023-07-20T19:00:36.103 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22d1a18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc108df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:36.103 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22d1a18700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc079e70 con 0x7f22cc106a00 2023-07-20T19:00:36.103 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc108df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:36.103 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc108df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57344/0 (socket says 172.21.15.61:57344) 2023-07-20T19:00:36.103 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.237+0000 7f22caffd700 1 -- 172.21.15.61:0/2769159178 learned_addr learned my addr 172.21.15.61:0/2769159178 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:36.104 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.238+0000 7f22caffd700 1 -- 172.21.15.61:0/2769159178 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc109330 con 0x7f22cc106a00 2023-07-20T19:00:36.104 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.238+0000 7f22caffd700 1 --2- 172.21.15.61:0/2769159178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc108df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f22c00098a0 tx=0x7f22c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82c20803166a549 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:36.104 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.238+0000 7f22c9ffb700 1 -- 172.21.15.61:0/2769159178 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22c0004160 con 0x7f22cc106a00 2023-07-20T19:00:36.105 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.238+0000 7f22c9ffb700 1 -- 172.21.15.61:0/2769159178 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22c00042c0 con 0x7f22cc106a00 2023-07-20T19:00:36.105 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.238+0000 7f22c9ffb700 1 -- 172.21.15.61:0/2769159178 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22c0011a00 con 0x7f22cc106a00 2023-07-20T19:00:36.105 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 -- 172.21.15.61:0/2769159178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 msgr2=0x7f22cc108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:36.105 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 --2- 172.21.15.61:0/2769159178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc108df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f22c00098a0 tx=0x7f22c0009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:36.106 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 -- 172.21.15.61:0/2769159178 shutdown_connections 2023-07-20T19:00:36.106 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 --2- 172.21.15.61:0/2769159178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc108df0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:36.106 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 -- 172.21.15.61:0/2769159178 >> 172.21.15.61:0/2769159178 conn(0x7f22cc100430 msgr2=0x7f22cc102850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:36.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 -- 172.21.15.61:0/2769159178 shutdown_connections 2023-07-20T19:00:36.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 -- 172.21.15.61:0/2769159178 wait complete. 2023-07-20T19:00:36.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.239+0000 7f22d1a18700 1 Processor -- start 2023-07-20T19:00:36.107 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22d1a18700 1 -- start start 2023-07-20T19:00:36.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22d1a18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc106360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:36.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22d1a18700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc1068a0 con 0x7f22cc106a00 2023-07-20T19:00:36.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc106360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:36.108 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc106360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57346/0 (socket says 172.21.15.61:57346) 2023-07-20T19:00:36.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22caffd700 1 -- 172.21.15.61:0/1219649467 learned_addr learned my addr 172.21.15.61:0/1219649467 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:36.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22caffd700 1 -- 172.21.15.61:0/1219649467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0009550 con 0x7f22cc106a00 2023-07-20T19:00:36.109 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22caffd700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc106360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009fb0 tx=0x7f22c0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:36.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.240+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22c0003930 con 0x7f22cc106a00 2023-07-20T19:00:36.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.241+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22c0003a90 con 0x7f22cc106a00 2023-07-20T19:00:36.110 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.241+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc1049b0 con 0x7f22cc106a00 2023-07-20T19:00:36.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.241+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22c0019610 con 0x7f22cc106a00 2023-07-20T19:00:36.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.241+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc104dd0 con 0x7f22cc106a00 2023-07-20T19:00:36.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.241+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f22c00198d0 con 0x7f22cc106a00 2023-07-20T19:00:36.111 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.242+0000 7f22d0a16700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f22b40386c0 0x7f22b403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:36.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.242+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f22c004c8f0 con 0x7f22cc106a00 2023-07-20T19:00:36.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.242+0000 7f22ca7fc700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f22b40386c0 0x7f22b403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:36.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.242+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22cc10a0d0 con 0x7f22cc106a00 2023-07-20T19:00:36.112 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.242+0000 7f22ca7fc700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f22b40386c0 0x7f22b403ab80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22bc006fd0 tx=0x7f22bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:36.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.247+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f22c0019bd0 con 0x7f22cc106a00 2023-07-20T19:00:36.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.287+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mon.0 v2:172.21.15.61:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22c0020ec0 con 0x7f22cc106a00 2023-07-20T19:00:36.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.638+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f22cc1018b0 con 0x7f22b40386c0 2023-07-20T19:00:36.113 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.647+0000 7f22d0a16700 1 -- 172.21.15.61:0/1219649467 <== mgr.14120 v2:172.21.15.61:6800/2098378944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f22cc1018b0 con 0x7f22b40386c0 2023-07-20T19:00:36.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f22b40386c0 msgr2=0x7f22b403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:36.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f22b40386c0 0x7f22b403ab80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22bc006fd0 tx=0x7f22bc006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:36.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 msgr2=0x7f22cc106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:36.114 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc106360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009fb0 tx=0x7f22c0011ad0 comp rx=0 tx=0).stop 2023-07-20T19:00:36.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 shutdown_connections 2023-07-20T19:00:36.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f22b40386c0 0x7f22b403ab80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:36.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 --2- 172.21.15.61:0/1219649467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22cc106a00 0x7f22cc106360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:36.115 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 >> 172.21.15.61:0/1219649467 conn(0x7f22cc100430 msgr2=0x7f22cc1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:36.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 shutdown_connections 2023-07-20T19:00:36.116 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:35.650+0000 7f22d1a18700 1 -- 172.21.15.61:0/1219649467 wait complete. 2023-07-20T19:00:36.339 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:36 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:36.339 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:36 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:36.339 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:36 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:37.446 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.506+0000 7f85ff284700 1 Processor -- start 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85ff284700 1 -- start start 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85ff284700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f8104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85ff284700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8079e30 con 0x7f85f8104840 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85fd020700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f8104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85fd020700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f8104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57350/0 (socket says 172.21.15.61:57350) 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85fd020700 1 -- 172.21.15.61:0/1957899698 learned_addr learned my addr 172.21.15.61:0/1957899698 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:37.447 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85fd020700 1 -- 172.21.15.61:0/1957899698 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f81051a0 con 0x7f85f8104840 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.507+0000 7f85fd020700 1 --2- 172.21.15.61:0/1957899698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f8104c60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85f40098a0 tx=0x7f85f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260309748ab80377 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85effff700 1 -- 172.21.15.61:0/1957899698 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85f4004160 con 0x7f85f8104840 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85effff700 1 -- 172.21.15.61:0/1957899698 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f85f40042c0 con 0x7f85f8104840 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85effff700 1 -- 172.21.15.61:0/1957899698 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85f40119b0 con 0x7f85f8104840 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 -- 172.21.15.61:0/1957899698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 msgr2=0x7f85f8104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 --2- 172.21.15.61:0/1957899698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f8104c60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85f40098a0 tx=0x7f85f4009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:37.448 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 -- 172.21.15.61:0/1957899698 shutdown_connections 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 --2- 172.21.15.61:0/1957899698 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f8104c60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 -- 172.21.15.61:0/1957899698 >> 172.21.15.61:0/1957899698 conn(0x7f85f8100400 msgr2=0x7f85f8102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 -- 172.21.15.61:0/1957899698 shutdown_connections 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.508+0000 7f85ff284700 1 -- 172.21.15.61:0/1957899698 wait complete. 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85ff284700 1 Processor -- start 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85ff284700 1 -- start start 2023-07-20T19:00:37.449 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85ff284700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f819be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:37.450 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85ff284700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f819c380 con 0x7f85f8104840 2023-07-20T19:00:37.450 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85fd020700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f819be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:37.451 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85fd020700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f819be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57352/0 (socket says 172.21.15.61:57352) 2023-07-20T19:00:37.451 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.509+0000 7f85fd020700 1 -- 172.21.15.61:0/3434290898 learned_addr learned my addr 172.21.15.61:0/3434290898 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:37.452 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85fd020700 1 -- 172.21.15.61:0/3434290898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4009550 con 0x7f85f8104840 2023-07-20T19:00:37.452 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85fd020700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f819be40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006af0 tx=0x7f85f4011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:37.452 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85f4003a40 con 0x7f85f8104840 2023-07-20T19:00:37.453 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f819c580 con 0x7f85f8104840 2023-07-20T19:00:37.453 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f85f4003ba0 con 0x7f85f8104840 2023-07-20T19:00:37.453 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85f40196c0 con 0x7f85f8104840 2023-07-20T19:00:37.453 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.510+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f819c9a0 con 0x7f85f8104840 2023-07-20T19:00:37.454 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.511+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f85f4019980 con 0x7f85f8104840 2023-07-20T19:00:37.454 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.511+0000 7f85ee7fc700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f85e40386c0 0x7f85e403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:37.454 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.511+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f85f404c6b0 con 0x7f85f8104840 2023-07-20T19:00:37.454 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.511+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f81956c0 con 0x7f85f8104840 2023-07-20T19:00:37.455 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.511+0000 7f85fc81f700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f85e40386c0 0x7f85e403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:37.455 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.512+0000 7f85fc81f700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f85e40386c0 0x7f85e403ab80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85e8006fd0 tx=0x7f85e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:37.456 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.518+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f85f40243f0 con 0x7f85f8104840 2023-07-20T19:00:37.456 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.884+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f85f8061da0 con 0x7f85f8104840 2023-07-20T19:00:37.456 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.887+0000 7f85ee7fc700 1 -- 172.21.15.61:0/3434290898 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f85f404a020 con 0x7f85f8104840 2023-07-20T19:00:37.456 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f85e40386c0 msgr2=0x7f85e403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:37.457 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f85e40386c0 0x7f85e403ab80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85e8006fd0 tx=0x7f85e8006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:37.457 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 msgr2=0x7f85f819be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:37.457 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f819be40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006af0 tx=0x7f85f4011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:37.458 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 shutdown_connections 2023-07-20T19:00:37.458 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f85e40386c0 0x7f85e403ab80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:37.458 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 --2- 172.21.15.61:0/3434290898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f8104840 0x7f85f819be40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:37.458 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 >> 172.21.15.61:0/3434290898 conn(0x7f85f8100400 msgr2=0x7f85f8190400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:37.459 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 shutdown_connections 2023-07-20T19:00:37.459 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:36.890+0000 7f85ff284700 1 -- 172.21.15.61:0/3434290898 wait complete. 2023-07-20T19:00:37.652 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:37 smithi061 ceph-mon[127454]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:37.653 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:37 smithi061 ceph-mon[127454]: Saving service mgr spec with placement count:2 2023-07-20T19:00:37.653 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:37 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3434290898' entity='client.admin' 2023-07-20T19:00:38.591 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.889+0000 7ff1bef91700 1 Processor -- start 2023-07-20T19:00:38.592 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.889+0000 7ff1bef91700 1 -- start start 2023-07-20T19:00:38.592 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.889+0000 7ff1bef91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b8104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:38.592 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.889+0000 7ff1bef91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b8079e30 con 0x7ff1b8104840 2023-07-20T19:00:38.592 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.890+0000 7ff1bcd2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b8104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:38.592 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.890+0000 7ff1bcd2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b8104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57356/0 (socket says 172.21.15.61:57356) 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.890+0000 7ff1bcd2d700 1 -- 172.21.15.61:0/442476228 learned_addr learned my addr 172.21.15.61:0/442476228 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.890+0000 7ff1bcd2d700 1 -- 172.21.15.61:0/442476228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b81051a0 con 0x7ff1b8104840 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.890+0000 7ff1bcd2d700 1 --2- 172.21.15.61:0/442476228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b8104c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0098a0 tx=0x7ff1ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1078fddeafc392 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1b77fe700 1 -- 172.21.15.61:0/442476228 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ac004160 con 0x7ff1b8104840 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1b77fe700 1 -- 172.21.15.61:0/442476228 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1ac0042c0 con 0x7ff1b8104840 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1b77fe700 1 -- 172.21.15.61:0/442476228 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ac0119b0 con 0x7ff1b8104840 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 -- 172.21.15.61:0/442476228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 msgr2=0x7ff1b8104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:38.593 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 --2- 172.21.15.61:0/442476228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b8104c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0098a0 tx=0x7ff1ac009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 -- 172.21.15.61:0/442476228 shutdown_connections 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 --2- 172.21.15.61:0/442476228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b8104c60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 -- 172.21.15.61:0/442476228 >> 172.21.15.61:0/442476228 conn(0x7ff1b8100400 msgr2=0x7ff1b8102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 -- 172.21.15.61:0/442476228 shutdown_connections 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.891+0000 7ff1bef91700 1 -- 172.21.15.61:0/442476228 wait complete. 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bef91700 1 Processor -- start 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bef91700 1 -- start start 2023-07-20T19:00:38.594 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bef91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bef91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b819c3c0 con 0x7ff1b8104840 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bcd2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bcd2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b819be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57358/0 (socket says 172.21.15.61:57358) 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.892+0000 7ff1bcd2d700 1 -- 172.21.15.61:0/1846508022 learned_addr learned my addr 172.21.15.61:0/1846508022 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1bcd2d700 1 -- 172.21.15.61:0/1846508022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac009550 con 0x7ff1b8104840 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1bcd2d700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b819be80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006af0 tx=0x7ff1ac011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:38.595 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ac003a40 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1ac003ba0 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b819c5c0 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1ac0196c0 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.893+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b819c9e0 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.894+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff1ac019930 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.894+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b8195730 con 0x7ff1b8104840 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.895+0000 7ff1b5ffb700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff1a8038710 0x7ff1a803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:38.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.895+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff1ac04c970 con 0x7ff1b8104840 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.895+0000 7ff1b7fff700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff1a8038710 0x7ff1a803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.895+0000 7ff1b7fff700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff1a8038710 0x7ff1a803abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4006fd0 tx=0x7ff1a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:37.900+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1ac01aa80 con 0x7ff1b8104840 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.263+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff1b802c970 con 0x7ff1b8104840 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.269+0000 7ff1b5ffb700 1 -- 172.21.15.61:0/1846508022 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff1ac01aa80 con 0x7ff1b8104840 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.271+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff1a8038710 msgr2=0x7ff1a803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:38.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.271+0000 7ff1bef91700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff1a8038710 0x7ff1a803abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4006fd0 tx=0x7ff1a4006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.271+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 msgr2=0x7ff1b819be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.271+0000 7ff1bef91700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b819be80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006af0 tx=0x7ff1ac011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.272+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 shutdown_connections 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.272+0000 7ff1bef91700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7ff1a8038710 0x7ff1a803abd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.272+0000 7ff1bef91700 1 --2- 172.21.15.61:0/1846508022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b8104840 0x7ff1b819be80 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006af0 tx=0x7ff1ac011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.272+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 >> 172.21.15.61:0/1846508022 conn(0x7ff1b8100400 msgr2=0x7ff1b8190400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.272+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 shutdown_connections 2023-07-20T19:00:38.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:38.272+0000 7ff1bef91700 1 -- 172.21.15.61:0/1846508022 wait complete. 2023-07-20T19:00:38.599 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling the dashboard module... 2023-07-20T19:00:38.617 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:38 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:38.618 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:38 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1846508022' entity='client.admin' 2023-07-20T19:00:38.618 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:38 smithi061 ceph-mon[127454]: from='mgr.14120 172.21.15.61:0/658828746' entity='mgr.x' 2023-07-20T19:00:39.713 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:39 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/409513998' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T19:00:39.714 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: ignoring --setuser ceph since I am not root 2023-07-20T19:00:39.714 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: ignoring --setgroup ceph since I am not root 2023-07-20T19:00:40.014 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:39.753+0000 7f7f331f1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T19:00:40.014 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:39.851+0000 7f7f331f1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T19:00:40.046 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.009+0000 7fc1f0ac8700 1 Processor -- start 2023-07-20T19:00:40.046 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.009+0000 7fc1f0ac8700 1 -- start start 2023-07-20T19:00:40.046 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.009+0000 7fc1f0ac8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec106860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:40.046 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.009+0000 7fc1f0ac8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec106e30 con 0x7fc1ec106440 2023-07-20T19:00:40.046 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.010+0000 7fc1ea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:40.046 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.010+0000 7fc1ea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57362/0 (socket says 172.21.15.61:57362) 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.010+0000 7fc1ea59c700 1 -- 172.21.15.61:0/112318937 learned_addr learned my addr 172.21.15.61:0/112318937 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.010+0000 7fc1ea59c700 1 -- 172.21.15.61:0/112318937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec106f70 con 0x7fc1ec106440 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.010+0000 7fc1ea59c700 1 --2- 172.21.15.61:0/112318937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec106860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40098a0 tx=0x7fc1e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aede3a2dc404da36 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.011+0000 7fc1e959a700 1 -- 172.21.15.61:0/112318937 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e4004160 con 0x7fc1ec106440 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.011+0000 7fc1e959a700 1 -- 172.21.15.61:0/112318937 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1e40042c0 con 0x7fc1ec106440 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.011+0000 7fc1e959a700 1 -- 172.21.15.61:0/112318937 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e40119b0 con 0x7fc1ec106440 2023-07-20T19:00:40.047 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.011+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/112318937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 msgr2=0x7fc1ec106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.011+0000 7fc1f0ac8700 1 --2- 172.21.15.61:0/112318937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec106860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40098a0 tx=0x7fc1e4009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.012+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/112318937 shutdown_connections 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.012+0000 7fc1f0ac8700 1 --2- 172.21.15.61:0/112318937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec106860 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.012+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/112318937 >> 172.21.15.61:0/112318937 conn(0x7fc1ec0ffbc0 msgr2=0x7fc1ec101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.012+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/112318937 shutdown_connections 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.012+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/112318937 wait complete. 2023-07-20T19:00:40.048 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1f0ac8700 1 Processor -- start 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1f0ac8700 1 -- start start 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1f0ac8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1f0ac8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1990d0 con 0x7fc1ec106440 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1ea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1ea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57364/0 (socket says 172.21.15.61:57364) 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1ea59c700 1 -- 172.21.15.61:0/409513998 learned_addr learned my addr 172.21.15.61:0/409513998 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:40.049 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1ea59c700 1 -- 172.21.15.61:0/409513998 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4009550 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.013+0000 7fc1ea59c700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec198b90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4003e60 tx=0x7fc1e4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.014+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e4003a60 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.014+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1e4003bc0 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.014+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec1992d0 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.014+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e40196c0 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.014+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec1996f0 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.015+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc1e4019980 con 0x7fc1ec106440 2023-07-20T19:00:40.050 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.015+0000 7fc1db7fe700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7fc1d403a900 0x7fc1d403cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.015+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc1e404c740 con 0x7fc1ec106440 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.016+0000 7fc1e9d9b700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7fc1d403a900 0x7fc1d403cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.016+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec106f70 con 0x7fc1ec106440 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.016+0000 7fc1e9d9b700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7fc1d403a900 0x7fc1d403cdc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0006fd0 tx=0x7fc1e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.021+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc1ec106f70 con 0x7fc1ec106440 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.495+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc1ec106f70 con 0x7fc1ec106440 2023-07-20T19:00:40.051 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.549+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fc1e402c8c0 con 0x7fc1ec106440 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.550+0000 7fc1db7fe700 1 -- 172.21.15.61:0/409513998 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc1e4010df0 con 0x7fc1ec106440 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7fc1d403a900 msgr2=0x7fc1d403cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7fc1d403a900 0x7fc1d403cdc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0006fd0 tx=0x7fc1e0006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 msgr2=0x7fc1ec198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec198b90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4003e60 tx=0x7fc1e4011a30 comp rx=0 tx=0).stop 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 shutdown_connections 2023-07-20T19:00:40.052 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7fc1d403a900 0x7fc1d403cdc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:40.053 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 --2- 172.21.15.61:0/409513998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1ec106440 0x7fc1ec198b90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:40.053 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.552+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 >> 172.21.15.61:0/409513998 conn(0x7fc1ec0ffbc0 msgr2=0x7fc1ec1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:40.053 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.553+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 shutdown_connections 2023-07-20T19:00:40.053 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:39.553+0000 7fc1f0ac8700 1 -- 172.21.15.61:0/409513998 wait complete. 2023-07-20T19:00:40.342 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:39.960+0000 7f7f331f1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T19:00:40.342 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:40.079+0000 7f7f331f1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T19:00:40.604 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:40.289+0000 7f7f331f1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T19:00:40.868 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/409513998' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T19:00:40.868 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-mon[127454]: mgrmap e9: x(active, since 18s) 2023-07-20T19:00:40.869 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:40.695+0000 7f7f331f1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T19:00:41.136 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:40.815+0000 7f7f331f1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T19:00:41.136 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:40.967+0000 7f7f331f1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T19:00:41.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:41.083+0000 7f7f331f1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T19:00:41.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:41.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T19:00:41.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T19:00:41.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T19:00:41.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T19:00:41.596 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.697+0000 7efe81fde700 1 Processor -- start 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe81fde700 1 -- start start 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe81fde700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe81fde700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c079eb0 con 0x7efe7c1069d0 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe7b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe7b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57378/0 (socket says 172.21.15.61:57378) 2023-07-20T19:00:41.597 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe7b7fe700 1 -- 172.21.15.61:0/4111830032 learned_addr learned my addr 172.21.15.61:0/4111830032 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:41.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe7b7fe700 1 -- 172.21.15.61:0/4111830032 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c109300 con 0x7efe7c1069d0 2023-07-20T19:00:41.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.698+0000 7efe7b7fe700 1 --2- 172.21.15.61:0/4111830032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c108dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0098a0 tx=0x7efe6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a579af2f1dc957 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:41.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe7a7fc700 1 -- 172.21.15.61:0/4111830032 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe6c004160 con 0x7efe7c1069d0 2023-07-20T19:00:41.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe7a7fc700 1 -- 172.21.15.61:0/4111830032 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe6c0042c0 con 0x7efe7c1069d0 2023-07-20T19:00:41.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe7a7fc700 1 -- 172.21.15.61:0/4111830032 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe6c0119b0 con 0x7efe7c1069d0 2023-07-20T19:00:41.598 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 -- 172.21.15.61:0/4111830032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 msgr2=0x7efe7c108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 --2- 172.21.15.61:0/4111830032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c108dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0098a0 tx=0x7efe6c009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 -- 172.21.15.61:0/4111830032 shutdown_connections 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 --2- 172.21.15.61:0/4111830032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c108dc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 -- 172.21.15.61:0/4111830032 >> 172.21.15.61:0/4111830032 conn(0x7efe7c100400 msgr2=0x7efe7c102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 -- 172.21.15.61:0/4111830032 shutdown_connections 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.699+0000 7efe81fde700 1 -- 172.21.15.61:0/4111830032 wait complete. 2023-07-20T19:00:41.599 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.700+0000 7efe81fde700 1 Processor -- start 2023-07-20T19:00:41.600 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.700+0000 7efe81fde700 1 -- start start 2023-07-20T19:00:41.600 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.700+0000 7efe81fde700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c106330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:41.600 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.700+0000 7efe81fde700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c106870 con 0x7efe7c1069d0 2023-07-20T19:00:41.600 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe7b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:41.600 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe7b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c106330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57380/0 (socket says 172.21.15.61:57380) 2023-07-20T19:00:41.600 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe7b7fe700 1 -- 172.21.15.61:0/205581207 learned_addr learned my addr 172.21.15.61:0/205581207 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe7b7fe700 1 -- 172.21.15.61:0/205581207 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c009550 con 0x7efe7c1069d0 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe7b7fe700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c106330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe6c006af0 tx=0x7efe6c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe6c003a40 con 0x7efe7c1069d0 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe6c003ba0 con 0x7efe7c1069d0 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c104980 con 0x7efe7c1069d0 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe6c0196c0 con 0x7efe7c1069d0 2023-07-20T19:00:41.601 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.701+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c104da0 con 0x7efe7c1069d0 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.702+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7efe6c019980 con 0x7efe7c1069d0 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.702+0000 7efe78ff9700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 0x7efe6403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.703+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efe6c04d630 con 0x7efe7c1069d0 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.703+0000 7efe7affd700 1 -- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 msgr2=0x7efe6403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.703+0000 7efe7affd700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 0x7efe6403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.703+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c10a140 con 0x7efe7c1069d0 2023-07-20T19:00:41.602 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.707+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efe6c0243f0 con 0x7efe7c1069d0 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.904+0000 7efe7affd700 1 -- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 msgr2=0x7efe6403abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:40.904+0000 7efe7affd700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 0x7efe6403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.182+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7efe7c19fc80 con 0x7efe7c1069d0 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.182+0000 7efe78ff9700 1 -- 172.21.15.61:0/205581207 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7efe6c04b030 con 0x7efe7c1069d0 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 msgr2=0x7efe6403abd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 0x7efe6403abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:41.603 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 msgr2=0x7efe7c106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c106330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe6c006af0 tx=0x7efe6c011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 shutdown_connections 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7efe64038710 0x7efe6403abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 --2- 172.21.15.61:0/205581207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe7c1069d0 0x7efe7c106330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 >> 172.21.15.61:0/205581207 conn(0x7efe7c100400 msgr2=0x7efe7c101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 shutdown_connections 2023-07-20T19:00:41.604 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:41.185+0000 7efe81fde700 1 -- 172.21.15.61:0/205581207 wait complete. 2023-07-20T19:00:41.605 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for the mgr to restart... 2023-07-20T19:00:41.605 INFO:teuthology.orchestra.run.smithi061.stdout:Waiting for mgr epoch 9... 2023-07-20T19:00:41.843 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:41 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/205581207' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T19:00:41.844 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:41.612+0000 7f7f331f1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T19:00:42.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:42.308+0000 7f7f331f1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T19:00:42.714 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:42.500+0000 7f7f331f1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T19:00:43.849 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:43.462+0000 7f7f331f1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:00:43.849 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:43.575+0000 7f7f331f1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T19:00:44.198 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:43.796+0000 7f7f331f1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:00:44.455 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:44.145+0000 7f7f331f1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T19:00:44.713 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:44.401+0000 7f7f331f1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T19:00:44.714 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:44.584+0000 7f7f331f1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T19:00:45.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:44.912+0000 7f7f331f1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T19:00:46.193 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:45.739+0000 7f7f331f1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T19:00:46.194 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:45.897+0000 7f7f331f1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T19:00:46.194 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:46.140+0000 7f7f331f1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T19:00:46.463 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:46.277+0000 7f7f331f1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T19:00:46.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:46.549+0000 7f7f331f1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T19:00:47.963 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:00:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:00:47.588+0000 7f7f331f1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T19:00:48.845 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: Active manager daemon x restarted 2023-07-20T19:00:48.845 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: Activating manager daemon x 2023-07-20T19:00:48.845 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T19:00:48.845 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: mgrmap e10: x(active, starting, since 0.00344434s) 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: Manager daemon x is now available 2023-07-20T19:00:48.846 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:48.847 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:48.847 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T19:00:48.847 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T19:00:49.056 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout { 2023-07-20T19:00:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T19:00:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T19:00:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout } 2023-07-20T19:00:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.031+0000 7f941bcc0700 1 Processor -- start 2023-07-20T19:00:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.031+0000 7f941bcc0700 1 -- start start 2023-07-20T19:00:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.031+0000 7f941bcc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.031+0000 7f941bcc0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941407a8b0 con 0x7f941407be80 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9419a5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9419a5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57388/0 (socket says 172.21.15.61:57388) 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9419a5c700 1 -- 172.21.15.61:0/3709290763 learned_addr learned my addr 172.21.15.61:0/3709290763 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9419a5c700 1 -- 172.21.15.61:0/3709290763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941407a9f0 con 0x7f941407be80 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9419a5c700 1 --2- 172.21.15.61:0/3709290763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941407a2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f94100098a0 tx=0x7f9410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e722d179a1a041 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9418a5a700 1 -- 172.21.15.61:0/3709290763 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9410004160 con 0x7f941407be80 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9418a5a700 1 -- 172.21.15.61:0/3709290763 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94100042c0 con 0x7f941407be80 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.032+0000 7f9418a5a700 1 -- 172.21.15.61:0/3709290763 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94100119b0 con 0x7f941407be80 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 -- 172.21.15.61:0/3709290763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 msgr2=0x7f941407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 --2- 172.21.15.61:0/3709290763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941407a2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f94100098a0 tx=0x7f9410009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 -- 172.21.15.61:0/3709290763 shutdown_connections 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 --2- 172.21.15.61:0/3709290763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941407a2e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 -- 172.21.15.61:0/3709290763 >> 172.21.15.61:0/3709290763 conn(0x7f9414101510 msgr2=0x7f9414103950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:49.059 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 -- 172.21.15.61:0/3709290763 shutdown_connections 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.033+0000 7f941bcc0700 1 -- 172.21.15.61:0/3709290763 wait complete. 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f941bcc0700 1 Processor -- start 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f941bcc0700 1 -- start start 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f941bcc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941419be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f941bcc0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941419c380 con 0x7f941407be80 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f9419a5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941419be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:49.060 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f9419a5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941419be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57390/0 (socket says 172.21.15.61:57390) 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.034+0000 7f9419a5c700 1 -- 172.21.15.61:0/2296206232 learned_addr learned my addr 172.21.15.61:0/2296206232 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f9419a5c700 1 -- 172.21.15.61:0/2296206232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410009550 con 0x7f941407be80 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f9419a5c700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941419be40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9410006af0 tx=0x7f9410011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9410003a40 con 0x7f941407be80 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9410003ba0 con 0x7f941407be80 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94100196c0 con 0x7f941407be80 2023-07-20T19:00:49.061 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941419c580 con 0x7f941407be80 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.035+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941419c9a0 con 0x7f941407be80 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.036+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f941001a430 con 0x7f941407be80 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.036+0000 7f940affd700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.036+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 --> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f940003b330 con 0x7f9400038760 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.036+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f941004c690 con 0x7f941407be80 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.036+0000 7f941925b700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 msgr2=0x7f940003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.036+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:00:49.062 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.237+0000 7f941925b700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 msgr2=0x7f940003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:49.063 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.237+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T19:00:49.063 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.638+0000 7f941925b700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 msgr2=0x7f940003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:49.063 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:42.638+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T19:00:49.063 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:43.439+0000 7f941925b700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 msgr2=0x7f940003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:49.063 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:43.439+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T19:00:49.063 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:45.040+0000 7f941925b700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 msgr2=0x7f940003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2098378944 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:45.040+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:47.603+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f9410019930 con 0x7f941407be80 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:47.604+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 msgr2=0x7f940003ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:47.604+0000 7f940affd700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.605+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f941004dfc0 con 0x7f941407be80 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.605+0000 7f940affd700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f940003bfb0 0x7f940003e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:49.064 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.605+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f940003b330 con 0x7f940003bfb0 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.606+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f940003bfb0 0x7f940003e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.606+0000 7f941925b700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f940003bfb0 0x7f940003e3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9404003a10 tx=0x7f9404009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.607+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f940003b330 con 0x7f940003bfb0 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.621+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f941419d180 con 0x7f940003bfb0 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.621+0000 7f940affd700 1 -- 172.21.15.61:0/2296206232 <== mgr.14152 v2:172.21.15.61:6800/3980156535 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f941419d180 con 0x7f940003bfb0 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.621+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f940003bfb0 msgr2=0x7f940003e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:49.065 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.621+0000 7f941bcc0700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f940003bfb0 0x7f940003e3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9404003a10 tx=0x7f9404009270 comp rx=0 tx=0).stop 2023-07-20T19:00:49.066 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.621+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 msgr2=0x7f941419be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:49.066 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.621+0000 7f941bcc0700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941419be40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9410006af0 tx=0x7f9410011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:49.066 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 shutdown_connections 2023-07-20T19:00:49.066 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f940003bfb0 0x7f940003e3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:49.066 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f941407be80 0x7f941419be40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:49.066 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 --2- 172.21.15.61:0/2296206232 >> [v2:172.21.15.61:6800/2098378944,v1:172.21.15.61:6801/2098378944] conn(0x7f9400038760 0x7f940003ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:49.067 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 >> 172.21.15.61:0/2296206232 conn(0x7f9414101510 msgr2=0x7f9414102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:49.067 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 shutdown_connections 2023-07-20T19:00:49.067 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:48.622+0000 7f941bcc0700 1 -- 172.21.15.61:0/2296206232 wait complete. 2023-07-20T19:00:49.067 INFO:teuthology.orchestra.run.smithi061.stdout:mgr epoch 9 is available 2023-07-20T19:00:49.067 INFO:teuthology.orchestra.run.smithi061.stdout:Generating a dashboard self-signed certificate... 2023-07-20T19:00:49.963 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:49 smithi061 ceph-mon[127454]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T19:00:49.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:49 smithi061 ceph-mon[127454]: mgrmap e11: x(active, since 1.00595s) 2023-07-20T19:00:49.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:49 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:50.553 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T19:00:50.553 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.556+0000 7fab6f288700 1 Processor -- start 2023-07-20T19:00:50.553 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.557+0000 7fab6f288700 1 -- start start 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.557+0000 7fab6f288700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab68106340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.557+0000 7fab6f288700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68106880 con 0x7fab68105f00 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.557+0000 7fab6d024700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab68106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.557+0000 7fab6d024700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab68106340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57444/0 (socket says 172.21.15.61:57444) 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.557+0000 7fab6d024700 1 -- 172.21.15.61:0/50918284 learned_addr learned my addr 172.21.15.61:0/50918284 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.558+0000 7fab6d024700 1 -- 172.21.15.61:0/50918284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab681069c0 con 0x7fab68105f00 2023-07-20T19:00:50.554 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.558+0000 7fab6d024700 1 --2- 172.21.15.61:0/50918284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab68106340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab640098a0 tx=0x7fab64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c960fa9256448341 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.558+0000 7fab5ffff700 1 -- 172.21.15.61:0/50918284 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab64004160 con 0x7fab68105f00 2023-07-20T19:00:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.558+0000 7fab5ffff700 1 -- 172.21.15.61:0/50918284 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab640042c0 con 0x7fab68105f00 2023-07-20T19:00:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.558+0000 7fab5ffff700 1 -- 172.21.15.61:0/50918284 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab640119b0 con 0x7fab68105f00 2023-07-20T19:00:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 -- 172.21.15.61:0/50918284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 msgr2=0x7fab68106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 --2- 172.21.15.61:0/50918284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab68106340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab640098a0 tx=0x7fab64009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 -- 172.21.15.61:0/50918284 shutdown_connections 2023-07-20T19:00:50.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 --2- 172.21.15.61:0/50918284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab68106340 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:50.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 -- 172.21.15.61:0/50918284 >> 172.21.15.61:0/50918284 conn(0x7fab68101550 msgr2=0x7fab68103970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:50.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 -- 172.21.15.61:0/50918284 shutdown_connections 2023-07-20T19:00:50.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 -- 172.21.15.61:0/50918284 wait complete. 2023-07-20T19:00:50.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 Processor -- start 2023-07-20T19:00:50.556 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.559+0000 7fab6f288700 1 -- start start 2023-07-20T19:00:50.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6f288700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab6807bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:50.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6f288700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6807c190 con 0x7fab68105f00 2023-07-20T19:00:50.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6d024700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab6807bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:50.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6d024700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab6807bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57446/0 (socket says 172.21.15.61:57446) 2023-07-20T19:00:50.557 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6d024700 1 -- 172.21.15.61:0/1260552084 learned_addr learned my addr 172.21.15.61:0/1260552084 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6d024700 1 -- 172.21.15.61:0/1260552084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64009550 con 0x7fab68105f00 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6d024700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab6807bc50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab64003e60 tx=0x7fab64011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab64003970 con 0x7fab68105f00 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab64003ad0 con 0x7fab68105f00 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.560+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6807a300 con 0x7fab68105f00 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.561+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab64019590 con 0x7fab68105f00 2023-07-20T19:00:50.558 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.561+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6807a780 con 0x7fab68105f00 2023-07-20T19:00:50.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.561+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fab64019840 con 0x7fab68105f00 2023-07-20T19:00:50.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.562+0000 7fab5e7fc700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab54038620 0x7fab5403aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:50.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.562+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab6404c7a0 con 0x7fab68105f00 2023-07-20T19:00:50.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.562+0000 7fab6c823700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab54038620 0x7fab5403aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:50.559 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.562+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab68061da0 con 0x7fab68105f00 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.562+0000 7fab6c823700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab54038620 0x7fab5403aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab58006fd0 tx=0x7fab58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.567+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab64019b90 con 0x7fab68105f00 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:49.950+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fab68103810 con 0x7fab54038620 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.146+0000 7fab5e7fc700 1 -- 172.21.15.61:0/1260552084 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fab68103810 con 0x7fab54038620 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab54038620 msgr2=0x7fab5403aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab54038620 0x7fab5403aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab58006fd0 tx=0x7fab58006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:50.560 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 msgr2=0x7fab6807bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab6807bc50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab64003e60 tx=0x7fab64011aa0 comp rx=0 tx=0).stop 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 shutdown_connections 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab54038620 0x7fab5403aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 --2- 172.21.15.61:0/1260552084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab68105f00 0x7fab6807bc50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 >> 172.21.15.61:0/1260552084 conn(0x7fab68101550 msgr2=0x7fab68102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 shutdown_connections 2023-07-20T19:00:50.561 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:50.150+0000 7fab6f288700 1 -- 172.21.15.61:0/1260552084 wait complete. 2023-07-20T19:00:50.562 INFO:teuthology.orchestra.run.smithi061.stdout:Creating initial admin user... 2023-07-20T19:00:50.963 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:48] ENGINE Bus STARTING 2023-07-20T19:00:50.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:48] ENGINE Serving on https://172.21.15.61:7150 2023-07-20T19:00:50.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:49] ENGINE Serving on http://172.21.15.61:8765 2023-07-20T19:00:50.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: [20/Jul/2023:19:00:49] ENGINE Bus STARTED 2023-07-20T19:00:50.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:50.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:50.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:51.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:51 smithi061 ceph-mon[127454]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:51.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:51 smithi061 ceph-mon[127454]: mgrmap e12: x(active, since 3s) 2023-07-20T19:00:52.209 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$eSVj.P.b9ZoOh44uuIue7O5ozHufaV4p4QDG9s5g6PVh3Z5IkGz0i", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689879651, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T19:00:52.209 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.019+0000 7ffb2b10f700 1 Processor -- start 2023-07-20T19:00:52.209 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.019+0000 7ffb2b10f700 1 -- start start 2023-07-20T19:00:52.209 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.019+0000 7ffb2b10f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb240712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.019+0000 7ffb2b10f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb240718c0 con 0x7ffb24070ed0 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.019+0000 7ffb28eab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb240712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb28eab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb240712f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57450/0 (socket says 172.21.15.61:57450) 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb28eab700 1 -- 172.21.15.61:0/342017955 learned_addr learned my addr 172.21.15.61:0/342017955 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb28eab700 1 -- 172.21.15.61:0/342017955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24071a00 con 0x7ffb24070ed0 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb28eab700 1 --2- 172.21.15.61:0/342017955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb240712f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb180098a0 tx=0x7ffb18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd896604b8ef2e1e server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:52.210 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb237fe700 1 -- 172.21.15.61:0/342017955 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb18004160 con 0x7ffb24070ed0 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb237fe700 1 -- 172.21.15.61:0/342017955 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb180042c0 con 0x7ffb24070ed0 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.020+0000 7ffb237fe700 1 -- 172.21.15.61:0/342017955 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb180119b0 con 0x7ffb24070ed0 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 -- 172.21.15.61:0/342017955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 msgr2=0x7ffb240712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 --2- 172.21.15.61:0/342017955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb240712f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb180098a0 tx=0x7ffb18009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 -- 172.21.15.61:0/342017955 shutdown_connections 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 --2- 172.21.15.61:0/342017955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb240712f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:52.211 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 -- 172.21.15.61:0/342017955 >> 172.21.15.61:0/342017955 conn(0x7ffb2406c720 msgr2=0x7ffb2406eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 -- 172.21.15.61:0/342017955 shutdown_connections 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.021+0000 7ffb2b10f700 1 -- 172.21.15.61:0/342017955 wait complete. 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb2b10f700 1 Processor -- start 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb2b10f700 1 -- start start 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb2b10f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb241a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb2b10f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb241a29d0 con 0x7ffb24070ed0 2023-07-20T19:00:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb28eab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb241a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb28eab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb241a2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57452/0 (socket says 172.21.15.61:57452) 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb28eab700 1 -- 172.21.15.61:0/3378100729 learned_addr learned my addr 172.21.15.61:0/3378100729 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.022+0000 7ffb28eab700 1 -- 172.21.15.61:0/3378100729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18009550 con 0x7ffb24070ed0 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.023+0000 7ffb28eab700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb241a2490 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb18003e60 tx=0x7ffb18011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.023+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb18003a60 con 0x7ffb24070ed0 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.023+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb18003bc0 con 0x7ffb24070ed0 2023-07-20T19:00:52.213 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.023+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb241a2bd0 con 0x7ffb24070ed0 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.023+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb180196c0 con 0x7ffb24070ed0 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.023+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb241a2ff0 con 0x7ffb24070ed0 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.024+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ffb18019930 con 0x7ffb24070ed0 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.024+0000 7ffb21ffb700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb0c038790 0x7ffb0c03ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.024+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffb1804cad0 con 0x7ffb24070ed0 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.024+0000 7ffb23fff700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb0c038790 0x7ffb0c03ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:52.214 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.024+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb24061da0 con 0x7ffb24070ed0 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.025+0000 7ffb23fff700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb0c038790 0x7ffb0c03ac50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c00ad30 tx=0x7ffb1c0093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.029+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffb18019e60 con 0x7ffb24070ed0 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.428+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ffb24061bc0 con 0x7ffb0c038790 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.675+0000 7ffb21ffb700 1 -- 172.21.15.61:0/3378100729 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ffb24061bc0 con 0x7ffb0c038790 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb0c038790 msgr2=0x7ffb0c03ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb0c038790 0x7ffb0c03ac50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c00ad30 tx=0x7ffb1c0093f0 comp rx=0 tx=0).stop 2023-07-20T19:00:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 msgr2=0x7ffb241a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb241a2490 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb18003e60 tx=0x7ffb18011a30 comp rx=0 tx=0).stop 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 shutdown_connections 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb0c038790 0x7ffb0c03ac50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 --2- 172.21.15.61:0/3378100729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb24070ed0 0x7ffb241a2490 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 >> 172.21.15.61:0/3378100729 conn(0x7ffb2406c720 msgr2=0x7ffb2406de40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 shutdown_connections 2023-07-20T19:00:52.216 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:51.678+0000 7ffb2b10f700 1 -- 172.21.15.61:0/3378100729 wait complete. 2023-07-20T19:00:52.217 INFO:teuthology.orchestra.run.smithi061.stdout:Fetching dashboard port number... 2023-07-20T19:00:52.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:52 smithi061 ceph-mon[127454]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:00:52.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:53.354 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T19:00:53.354 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.592+0000 7f2118784700 1 Processor -- start 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2118784700 1 -- start start 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2118784700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f2110104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2118784700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2110079e30 con 0x7f2110104840 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2116520700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f2110104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2116520700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f2110104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57456/0 (socket says 172.21.15.61:57456) 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2116520700 1 -- 172.21.15.61:0/3666199895 learned_addr learned my addr 172.21.15.61:0/3666199895 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2116520700 1 -- 172.21.15.61:0/3666199895 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21101051a0 con 0x7f2110104840 2023-07-20T19:00:53.355 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.593+0000 7f2116520700 1 --2- 172.21.15.61:0/3666199895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f2110104c60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f210c0098a0 tx=0x7f210c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59363b508a0b8b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f211551e700 1 -- 172.21.15.61:0/3666199895 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c004160 con 0x7f2110104840 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f211551e700 1 -- 172.21.15.61:0/3666199895 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f210c0042c0 con 0x7f2110104840 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f211551e700 1 -- 172.21.15.61:0/3666199895 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c0119b0 con 0x7f2110104840 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 -- 172.21.15.61:0/3666199895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 msgr2=0x7f2110104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 --2- 172.21.15.61:0/3666199895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f2110104c60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f210c0098a0 tx=0x7f210c009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 -- 172.21.15.61:0/3666199895 shutdown_connections 2023-07-20T19:00:53.356 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 --2- 172.21.15.61:0/3666199895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f2110104c60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 -- 172.21.15.61:0/3666199895 >> 172.21.15.61:0/3666199895 conn(0x7f2110100400 msgr2=0x7f2110102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 -- 172.21.15.61:0/3666199895 shutdown_connections 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.594+0000 7f2118784700 1 -- 172.21.15.61:0/3666199895 wait complete. 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2118784700 1 Processor -- start 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2118784700 1 -- start start 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2118784700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f211019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:53.357 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2118784700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211019c3c0 con 0x7f2110104840 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2116520700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f211019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2116520700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f211019be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57458/0 (socket says 172.21.15.61:57458) 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.595+0000 7f2116520700 1 -- 172.21.15.61:0/4293883377 learned_addr learned my addr 172.21.15.61:0/4293883377 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f2116520700 1 -- 172.21.15.61:0/4293883377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c009550 con 0x7f2110104840 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f2116520700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f211019be80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f210c006af0 tx=0x7f210c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c003a40 con 0x7f2110104840 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f210c003ba0 con 0x7f2110104840 2023-07-20T19:00:53.358 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211019c5c0 con 0x7f2110104840 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c0196c0 con 0x7f2110104840 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.596+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211019c9e0 con 0x7f2110104840 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.597+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f210c01aa80 con 0x7f2110104840 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.597+0000 7f21037fe700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f20fc038760 0x7f20fc03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.597+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f210c04cb80 con 0x7f2110104840 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.597+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2110195730 con 0x7f2110104840 2023-07-20T19:00:53.359 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.597+0000 7f2115d1f700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f20fc038760 0x7f20fc03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.598+0000 7f2115d1f700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f20fc038760 0x7f20fc03ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2104006fd0 tx=0x7f2104006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.602+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f210c02d430 con 0x7f2110104840 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.957+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f211002cdf0 con 0x7f2110104840 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.957+0000 7f21037fe700 1 -- 172.21.15.61:0/4293883377 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f211002cdf0 con 0x7f2110104840 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f20fc038760 msgr2=0x7f20fc03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f20fc038760 0x7f20fc03ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2104006fd0 tx=0x7f2104006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:53.360 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 msgr2=0x7f211019be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f211019be80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f210c006af0 tx=0x7f210c011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 shutdown_connections 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f20fc038760 0x7f20fc03ac20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 --2- 172.21.15.61:0/4293883377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2110104840 0x7f211019be80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 >> 172.21.15.61:0/4293883377 conn(0x7f2110100400 msgr2=0x7f2110190400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 shutdown_connections 2023-07-20T19:00:53.361 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:52.959+0000 7f2118784700 1 -- 172.21.15.61:0/4293883377 wait complete. 2023-07-20T19:00:53.362 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T19:00:53.363 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout disabled 2023-07-20T19:00:53.370 INFO:teuthology.orchestra.run.smithi061.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T19:00:53.370 INFO:teuthology.orchestra.run.smithi061.stdout:systemctl: stdout inactive 2023-07-20T19:00:53.370 INFO:teuthology.orchestra.run.smithi061.stdout:firewalld.service is not enabled 2023-07-20T19:00:53.370 INFO:teuthology.orchestra.run.smithi061.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T19:00:53.372 INFO:teuthology.orchestra.run.smithi061.stdout:Ceph Dashboard is now available at: 2023-07-20T19:00:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: URL: https://smithi061.front.sepia.ceph.com:8443/ 2023-07-20T19:00:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: User: admin 2023-07-20T19:00:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: Password: c9567z58x3 2023-07-20T19:00:53.373 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:53.373 INFO:teuthology.orchestra.run.smithi061.stdout:Saving cluster configuration to /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config directory 2023-07-20T19:00:53.373 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling autotune for osd_memory_target 2023-07-20T19:00:53.963 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:53 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/4293883377' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T19:00:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007f232700 1 Processor -- start 2023-07-20T19:00:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007f232700 1 -- start start 2023-07-20T19:00:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007f232700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007f232700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078079e70 con 0x7f0078106a10 2023-07-20T19:00:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007cfce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007cfce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57466/0 (socket says 172.21.15.61:57466) 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.823+0000 7f007cfce700 1 -- 172.21.15.61:0/1211289244 learned_addr learned my addr 172.21.15.61:0/1211289244 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f007cfce700 1 -- 172.21.15.61:0/1211289244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0078109340 con 0x7f0078106a10 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f007cfce700 1 --2- 172.21.15.61:0/1211289244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078108e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f006c0098a0 tx=0x7f006c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7502721815cc1e server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f00777fe700 1 -- 172.21.15.61:0/1211289244 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f006c004160 con 0x7f0078106a10 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f00777fe700 1 -- 172.21.15.61:0/1211289244 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f006c0042c0 con 0x7f0078106a10 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f00777fe700 1 -- 172.21.15.61:0/1211289244 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f006c0119b0 con 0x7f0078106a10 2023-07-20T19:00:54.618 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f007f232700 1 -- 172.21.15.61:0/1211289244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 msgr2=0x7f0078108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.824+0000 7f007f232700 1 --2- 172.21.15.61:0/1211289244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078108e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f006c0098a0 tx=0x7f006c009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 -- 172.21.15.61:0/1211289244 shutdown_connections 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 --2- 172.21.15.61:0/1211289244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078108e00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 -- 172.21.15.61:0/1211289244 >> 172.21.15.61:0/1211289244 conn(0x7f0078100400 msgr2=0x7f0078102840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 -- 172.21.15.61:0/1211289244 shutdown_connections 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 -- 172.21.15.61:0/1211289244 wait complete. 2023-07-20T19:00:54.619 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 Processor -- start 2023-07-20T19:00:54.620 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 -- start start 2023-07-20T19:00:54.620 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078106370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:54.620 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.825+0000 7f007f232700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00781068b0 con 0x7f0078106a10 2023-07-20T19:00:54.620 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007cfce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078106370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:54.622 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007cfce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078106370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57468/0 (socket says 172.21.15.61:57468) 2023-07-20T19:00:54.622 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007cfce700 1 -- 172.21.15.61:0/370096642 learned_addr learned my addr 172.21.15.61:0/370096642 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:54.622 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007cfce700 1 -- 172.21.15.61:0/370096642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c009550 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007cfce700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078106370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f006c006af0 tx=0x7f006c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f006c003a40 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00781049c0 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f006c003ba0 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f006c0196c0 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.826+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0078104de0 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.827+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f006c019980 con 0x7f0078106a10 2023-07-20T19:00:54.623 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.827+0000 7f0075ffb700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0068038710 0x7f006803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.827+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f006c04cc30 con 0x7f0078106a10 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.828+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007810a120 con 0x7f0078106a10 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.828+0000 7f0077fff700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0068038710 0x7f006803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.828+0000 7f0077fff700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0068038710 0x7f006803abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:53.832+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f006c0243f0 con 0x7f0078106a10 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.187+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f007810a2e0 con 0x7f0078106a10 2023-07-20T19:00:54.624 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.188+0000 7f0075ffb700 1 -- 172.21.15.61:0/370096642 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f006c04a020 con 0x7f0078106a10 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.190+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0068038710 msgr2=0x7f006803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.190+0000 7f007f232700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0068038710 0x7f006803abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.190+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 msgr2=0x7f0078106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.190+0000 7f007f232700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078106370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f006c006af0 tx=0x7f006c011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.191+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 shutdown_connections 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.191+0000 7f007f232700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0068038710 0x7f006803abd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.191+0000 7f007f232700 1 --2- 172.21.15.61:0/370096642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0078106a10 0x7f0078106370 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:54.625 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.191+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 >> 172.21.15.61:0/370096642 conn(0x7f0078100400 msgr2=0x7f0078101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:54.626 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.191+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 shutdown_connections 2023-07-20T19:00:54.626 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:54.191+0000 7f007f232700 1 -- 172.21.15.61:0/370096642 wait complete. 2023-07-20T19:00:55.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:55 smithi061 ceph-mon[127454]: mgrmap e13: x(active, since 7s) 2023-07-20T19:00:56.068 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.039+0000 7fec925c6700 1 Processor -- start 2023-07-20T19:00:56.068 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.039+0000 7fec925c6700 1 -- start start 2023-07-20T19:00:56.069 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.039+0000 7fec925c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c104670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:56.069 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.039+0000 7fec925c6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c104c40 con 0x7fec8c104250 2023-07-20T19:00:56.069 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:56.069 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57472/0 (socket says 172.21.15.61:57472) 2023-07-20T19:00:56.069 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8bfff700 1 -- 172.21.15.61:0/2624349046 learned_addr learned my addr 172.21.15.61:0/2624349046 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:56.070 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8bfff700 1 -- 172.21.15.61:0/2624349046 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c1071e0 con 0x7fec8c104250 2023-07-20T19:00:56.070 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8bfff700 1 --2- 172.21.15.61:0/2624349046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c104670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0098a0 tx=0x7fec7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabdcf847c128cb9 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:56.070 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8affd700 1 -- 172.21.15.61:0/2624349046 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c004160 con 0x7fec8c104250 2023-07-20T19:00:56.071 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8affd700 1 -- 172.21.15.61:0/2624349046 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec7c0042c0 con 0x7fec8c104250 2023-07-20T19:00:56.071 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.040+0000 7fec8affd700 1 -- 172.21.15.61:0/2624349046 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c0119b0 con 0x7fec8c104250 2023-07-20T19:00:56.071 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 -- 172.21.15.61:0/2624349046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 msgr2=0x7fec8c104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:56.071 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 --2- 172.21.15.61:0/2624349046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c104670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0098a0 tx=0x7fec7c009bb0 comp rx=0 tx=0).stop 2023-07-20T19:00:56.072 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 -- 172.21.15.61:0/2624349046 shutdown_connections 2023-07-20T19:00:56.072 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 --2- 172.21.15.61:0/2624349046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c104670 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:56.072 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 -- 172.21.15.61:0/2624349046 >> 172.21.15.61:0/2624349046 conn(0x7fec8c0ffbc0 msgr2=0x7fec8c101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:56.073 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 -- 172.21.15.61:0/2624349046 shutdown_connections 2023-07-20T19:00:56.073 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.041+0000 7fec925c6700 1 -- 172.21.15.61:0/2624349046 wait complete. 2023-07-20T19:00:56.073 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec925c6700 1 Processor -- start 2023-07-20T19:00:56.073 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec925c6700 1 -- start start 2023-07-20T19:00:56.074 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec925c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:56.074 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec925c6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c19a190 con 0x7fec8c104250 2023-07-20T19:00:56.074 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:56.075 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57474/0 (socket says 172.21.15.61:57474) 2023-07-20T19:00:56.075 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec8bfff700 1 -- 172.21.15.61:0/2551730504 learned_addr learned my addr 172.21.15.61:0/2551730504 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:56.075 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec8bfff700 1 -- 172.21.15.61:0/2551730504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009550 con 0x7fec8c104250 2023-07-20T19:00:56.075 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.042+0000 7fec8bfff700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c199c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006af0 tx=0x7fec7c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:56.075 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.043+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c003a40 con 0x7fec8c104250 2023-07-20T19:00:56.076 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.043+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec7c003ba0 con 0x7fec8c104250 2023-07-20T19:00:56.076 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.043+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec7c019690 con 0x7fec8c104250 2023-07-20T19:00:56.076 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.043+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c19a390 con 0x7fec8c104250 2023-07-20T19:00:56.076 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.043+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c19a7b0 con 0x7fec8c104250 2023-07-20T19:00:56.076 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.044+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fec7c01aa60 con 0x7fec8c104250 2023-07-20T19:00:56.076 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.044+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c10a410 con 0x7fec8c104250 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.044+0000 7fec897fa700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fec74038800 0x7fec7403acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.044+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec7c04cb60 con 0x7fec8c104250 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.044+0000 7fec8b7fe700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fec74038800 0x7fec7403acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.045+0000 7fec8b7fe700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fec74038800 0x7fec7403acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.049+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fec7c0237a0 con 0x7fec8c104250 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.617+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fec8c10a600 con 0x7fec8c104250 2023-07-20T19:00:56.077 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.622+0000 7fec897fa700 1 -- 172.21.15.61:0/2551730504 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fec7c023390 con 0x7fec8c104250 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fec74038800 msgr2=0x7fec7403acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fec74038800 0x7fec7403acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 msgr2=0x7fec8c199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c199c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006af0 tx=0x7fec7c011a10 comp rx=0 tx=0).stop 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 shutdown_connections 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fec74038800 0x7fec7403acc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:56.078 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 --2- 172.21.15.61:0/2551730504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec8c104250 0x7fec8c199c50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 >> 172.21.15.61:0/2551730504 conn(0x7fec8c0ffbc0 msgr2=0x7fec8c100930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 shutdown_connections 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr 2023-07-20T19:00:55.624+0000 7fec925c6700 1 -- 172.21.15.61:0/2551730504 wait complete. 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.079 INFO:teuthology.orchestra.run.smithi061.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:00:56.080 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.080 INFO:teuthology.orchestra.run.smithi061.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T19:00:56.080 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.080 INFO:teuthology.orchestra.run.smithi061.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T19:00:56.080 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.080 INFO:teuthology.orchestra.run.smithi061.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T19:00:56.081 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.081 INFO:teuthology.orchestra.run.smithi061.stdout: ceph telemetry on 2023-07-20T19:00:56.081 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.081 INFO:teuthology.orchestra.run.smithi061.stdout:For more information see: 2023-07-20T19:00:56.081 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.081 INFO:teuthology.orchestra.run.smithi061.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T19:00:56.082 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:00:56.082 INFO:teuthology.orchestra.run.smithi061.stdout:Bootstrap complete. 2023-07-20T19:00:56.098 INFO:tasks.cephadm:Fetching config... 2023-07-20T19:00:56.098 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:00:56.099 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T19:00:56.120 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T19:00:56.120 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:00:56.121 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T19:00:56.177 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T19:00:56.178 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:00:56.178 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T19:00:56.246 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T19:00:56.247 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:00:56.248 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T19:00:56.303 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T19:00:56.303 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T19:00:56.381 INFO:teuthology.orchestra.run.smithi061.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:00:56.396 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T19:00:56.443 INFO:teuthology.orchestra.run.smithi149.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:00:56.455 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T19:00:56.501 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC840QGUkVq5mO5tsvUx89VeEwS00zwPlB97LzlW4Iv9INWl2wPB4NDctBuRHbo3nDguxLFN6uvUa1Znz20qTQSsdNkhVB+pJSKCImcYlih1VBXgER3AuJ87JVYNbU+lFCvUIzHVN3E40NhoL9+t2AkrTLmEL7REbGzxHZN6WwVVStORwnpt1iXlwohwmEpAMDNqRwpMsdmEe9+5WRBSwfXkLpOvwtjQpjk6RGY1X/E/IO3g22gL4RCPnAFvhRR+qaUZCgQWVq70vAjoEmh49Qulg6aZDFZjd8hut3bnQDjHmiv2uFS5TIfRGqhBscutKj6xsI8nh3i5B+BxGgVHZfhwH7QDaAJ6wU137cw2hBiGsnPKnV0UftIaFQHknuLBTeCp9P4dcmPD2meWyvziwFG8ivuveY5diPR5yB/sM4qS+MPVIYEPs/cV7FHbL2i6fl3JWF7Qh7gcR/ggw12ky3uiAVxAWquvX7PDltMO7+evhxLnnXvgmkmYrbxl7c0KqM= ceph-7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:00:56.514 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T19:00:56.963 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:56 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2551730504' entity='client.admin' 2023-07-20T19:00:57.448 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:00:58.894 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 -- 172.21.15.61:0/1021351505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 msgr2=0x7f49a40fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:58.894 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 --2- 172.21.15.61:0/1021351505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a40fed10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f499400aa20 tx=0x7f499400ad30 comp rx=0 tx=0).stop 2023-07-20T19:00:58.894 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 -- 172.21.15.61:0/1021351505 shutdown_connections 2023-07-20T19:00:58.895 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 --2- 172.21.15.61:0/1021351505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a40fed10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:58.895 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 -- 172.21.15.61:0/1021351505 >> 172.21.15.61:0/1021351505 conn(0x7f49a40f9e90 msgr2=0x7f49a40fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:58.895 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 -- 172.21.15.61:0/1021351505 shutdown_connections 2023-07-20T19:00:58.895 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.841+0000 7f49a9431700 1 -- 172.21.15.61:0/1021351505 wait complete. 2023-07-20T19:00:58.897 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.842+0000 7f49a9431700 1 Processor -- start 2023-07-20T19:00:58.897 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.842+0000 7f49a9431700 1 -- start start 2023-07-20T19:00:58.897 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.842+0000 7f49a9431700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a4193110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:58.897 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.842+0000 7f49a9431700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4193650 con 0x7f49a40fe8f0 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.842+0000 7f49a2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a4193110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f49a2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a4193110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57480/0 (socket says 172.21.15.61:57480) 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f49a2ffd700 1 -- 172.21.15.61:0/3907641254 learned_addr learned my addr 172.21.15.61:0/3907641254 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f49a2ffd700 1 -- 172.21.15.61:0/3907641254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499400a700 con 0x7f49a40fe8f0 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f49a2ffd700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a4193110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f499400d010 tx=0x7f4994004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4994005470 con 0x7f49a40fe8f0 2023-07-20T19:00:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49940055d0 con 0x7f49a40fe8f0 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a4193850 con 0x7f49a40fe8f0 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f499401f740 con 0x7f49a40fe8f0 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.843+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a4193cf0 con 0x7f49a40fe8f0 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.844+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4994016470 con 0x7f49a40fe8f0 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.844+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a418cb70 con 0x7f49a40fe8f0 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.844+0000 7f498bfff700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f498c038800 0x7f498c03acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:00:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.844+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4994012070 con 0x7f49a40fe8f0 2023-07-20T19:00:58.900 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.844+0000 7f49a27fc700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f498c038800 0x7f498c03acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:00:58.900 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.845+0000 7f49a27fc700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f498c038800 0x7f498c03acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4998006fd0 tx=0x7f4998006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:00:58.901 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:58.849+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f499401cb60 con 0x7f49a40fe8f0 2023-07-20T19:00:59.100 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.047+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f49a402c970 con 0x7f49a40fe8f0 2023-07-20T19:00:59.103 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.051+0000 7f498bfff700 1 -- 172.21.15.61:0/3907641254 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f499401b2d0 con 0x7f49a40fe8f0 2023-07-20T19:00:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f498c038800 msgr2=0x7f498c03acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f498c038800 0x7f498c03acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4998006fd0 tx=0x7f4998006e40 comp rx=0 tx=0).stop 2023-07-20T19:00:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 msgr2=0x7f49a4193110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:00:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a4193110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f499400d010 tx=0x7f4994004ee0 comp rx=0 tx=0).stop 2023-07-20T19:00:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 shutdown_connections 2023-07-20T19:00:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f498c038800 0x7f498c03acc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:59.107 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 --2- 172.21.15.61:0/3907641254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49a40fe8f0 0x7f49a4193110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:00:59.107 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 >> 172.21.15.61:0/3907641254 conn(0x7f49a40f9e90 msgr2=0x7f49a40fabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:00:59.107 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 shutdown_connections 2023-07-20T19:00:59.107 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:00:59.053+0000 7f49a9431700 1 -- 172.21.15.61:0/3907641254 wait complete. 2023-07-20T19:00:59.621 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T19:00:59.622 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T19:00:59.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:59.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:59.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi061", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:00:59.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:59.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:59.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:00:59.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3907641254' entity='client.admin' 2023-07-20T19:00:59.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:00:59.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:00:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:00.182 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 -- 172.21.15.61:0/4280699647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 msgr2=0x7f74380ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 --2- 172.21.15.61:0/4280699647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380ff580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f742c009910 tx=0x7f742c009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 -- 172.21.15.61:0/4280699647 shutdown_connections 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 --2- 172.21.15.61:0/4280699647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380ff580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 -- 172.21.15.61:0/4280699647 >> 172.21.15.61:0/4280699647 conn(0x7f74380fa700 msgr2=0x7f74380fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 -- 172.21.15.61:0/4280699647 shutdown_connections 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.214+0000 7f743ed67700 1 -- 172.21.15.61:0/4280699647 wait complete. 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743ed67700 1 Processor -- start 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743ed67700 1 -- start start 2023-07-20T19:01:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743ed67700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380684f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743ed67700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381980a0 con 0x7f74380ff160 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743cb03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743cb03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57486/0 (socket says 172.21.15.61:57486) 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743cb03700 1 -- 172.21.15.61:0/4132480384 learned_addr learned my addr 172.21.15.61:0/4132480384 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.215+0000 7f743cb03700 1 -- 172.21.15.61:0/4132480384 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0095f0 con 0x7f74380ff160 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.216+0000 7f743cb03700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380684f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f742c006010 tx=0x7f742c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.216+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f742c01d070 con 0x7f74380ff160 2023-07-20T19:01:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.216+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f742c0044e0 con 0x7f74380ff160 2023-07-20T19:01:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.216+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f742c0039b0 con 0x7f74380ff160 2023-07-20T19:01:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.216+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74381981e0 con 0x7f74380ff160 2023-07-20T19:01:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.216+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7438198620 con 0x7f74380ff160 2023-07-20T19:01:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.217+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f742c00baf0 con 0x7f74380ff160 2023-07-20T19:01:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.217+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7438061e10 con 0x7f74380ff160 2023-07-20T19:01:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.217+0000 7f7435ffb700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f74280387b0 0x7f742803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.217+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f742c04c810 con 0x7f74380ff160 2023-07-20T19:01:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.217+0000 7f7437fff700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f74280387b0 0x7f742803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.218+0000 7f7437fff700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f74280387b0 0x7f742803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7424006fd0 tx=0x7f7424006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.222+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7438061e10 con 0x7f74380ff160 2023-07-20T19:01:01.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.424+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7438061e30 con 0x7f74280387b0 2023-07-20T19:01:01.480 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.428+0000 7f7435ffb700 1 -- 172.21.15.61:0/4132480384 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7424009e20 con 0x7f74280387b0 2023-07-20T19:01:01.482 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f74280387b0 msgr2=0x7f742803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f74280387b0 0x7f742803ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7424006fd0 tx=0x7f7424006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 msgr2=0x7f74380684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380684f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f742c006010 tx=0x7f742c0057c0 comp rx=0 tx=0).stop 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 shutdown_connections 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f74280387b0 0x7f742803ac70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 --2- 172.21.15.61:0/4132480384 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74380ff160 0x7f74380684f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:01.483 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 >> 172.21.15.61:0/4132480384 conn(0x7f74380fa700 msgr2=0x7f74380fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:01.484 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 shutdown_connections 2023-07-20T19:01:01.484 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:01.430+0000 7f743ed67700 1 -- 172.21.15.61:0/4132480384 wait complete. 2023-07-20T19:01:01.945 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi149 2023-07-20T19:01:01.946 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T19:01:01.946 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.conf 2023-07-20T19:01:01.967 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T19:01:01.967 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:02.025 INFO:tasks.cephadm:Adding host smithi149 to orchestrator... 2023-07-20T19:01:02.026 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch host add smithi149 2023-07-20T19:01:02.601 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:02.799 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:02 smithi061 ceph-mon[127454]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:02.799 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:02.799 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:02.799 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:02.799 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:02.799 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:02 smithi061 ceph-mon[127454]: Updating smithi061:/etc/ceph/ceph.conf 2023-07-20T19:01:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 -- 172.21.15.61:0/1196838003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 msgr2=0x7fcdc8100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 --2- 172.21.15.61:0/1196838003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc8100830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8009910 tx=0x7fcdb8009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 -- 172.21.15.61:0/1196838003 shutdown_connections 2023-07-20T19:01:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 --2- 172.21.15.61:0/1196838003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc8100830 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 -- 172.21.15.61:0/1196838003 >> 172.21.15.61:0/1196838003 conn(0x7fcdc80fbd00 msgr2=0x7fcdc80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 -- 172.21.15.61:0/1196838003 shutdown_connections 2023-07-20T19:01:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.751+0000 7fcdcc833700 1 -- 172.21.15.61:0/1196838003 wait complete. 2023-07-20T19:01:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdcc833700 1 Processor -- start 2023-07-20T19:01:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdcc833700 1 -- start start 2023-07-20T19:01:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdcc833700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc81956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdcc833700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8195c00 con 0x7fcdc8100450 2023-07-20T19:01:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdc659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc81956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdc659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc81956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57492/0 (socket says 172.21.15.61:57492) 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdc659c700 1 -- 172.21.15.61:0/2492230216 learned_addr learned my addr 172.21.15.61:0/2492230216 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdc659c700 1 -- 172.21.15.61:0/2492230216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb80095f0 con 0x7fcdc8100450 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.752+0000 7fcdc659c700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc81956c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8006010 tx=0x7fcdb8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.753+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb801d070 con 0x7fcdc8100450 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.753+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc8195e00 con 0x7fcdc8100450 2023-07-20T19:01:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.753+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdb800bc10 con 0x7fcdc8100450 2023-07-20T19:01:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.753+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb8003910 con 0x7fcdc8100450 2023-07-20T19:01:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.753+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc8196220 con 0x7fcdc8100450 2023-07-20T19:01:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.754+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc8061da0 con 0x7fcdc8100450 2023-07-20T19:01:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.754+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fcdb800bd80 con 0x7fcdc8100450 2023-07-20T19:01:03.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.754+0000 7fcdb77fe700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcdb00387b0 0x7fcdb003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:03.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.754+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcdb804c540 con 0x7fcdc8100450 2023-07-20T19:01:03.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.754+0000 7fcdc5d9b700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcdb00387b0 0x7fcdb003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:03.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.755+0000 7fcdc5d9b700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcdb00387b0 0x7fcdb003ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc006fd0 tx=0x7fcdbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:03.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.759+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcdb8029360 con 0x7fcdc8100450 2023-07-20T19:01:03.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:03 smithi061 ceph-mon[127454]: Updating smithi061:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:03.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:03 smithi061 ceph-mon[127454]: Updating smithi061:/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:03.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:03 smithi061 ceph-mon[127454]: Updating smithi061:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T19:01:03.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:03 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:03.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:03 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:03.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:03 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:04.012 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:03.960+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc8108210 con 0x7fcdb00387b0 2023-07-20T19:01:05.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:04 smithi061 ceph-mon[127454]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:05.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:04 smithi061 ceph-mon[127454]: Deploying cephadm binary to smithi149 2023-07-20T19:01:06.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.788+0000 7fcdb77fe700 1 -- 172.21.15.61:0/2492230216 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcdc8108210 con 0x7fcdb00387b0 2023-07-20T19:01:06.841 INFO:teuthology.orchestra.run.smithi061.stdout:Added host 'smithi149' with addr '172.21.15.149' 2023-07-20T19:01:06.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.790+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcdb00387b0 msgr2=0x7fcdb003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:06.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.790+0000 7fcdcc833700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcdb00387b0 0x7fcdb003ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc006fd0 tx=0x7fcdbc006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:06.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.790+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 msgr2=0x7fcdc81956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.790+0000 7fcdcc833700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc81956c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8006010 tx=0x7fcdb8004ff0 comp rx=0 tx=0).stop 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.791+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 shutdown_connections 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.791+0000 7fcdcc833700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcdb00387b0 0x7fcdb003ac70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.791+0000 7fcdcc833700 1 --2- 172.21.15.61:0/2492230216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc8100450 0x7fcdc81956c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.791+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 >> 172.21.15.61:0/2492230216 conn(0x7fcdc80fbd00 msgr2=0x7fcdc81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.791+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 shutdown_connections 2023-07-20T19:01:06.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:06.791+0000 7fcdcc833700 1 -- 172.21.15.61:0/2492230216 wait complete. 2023-07-20T19:01:07.169 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T19:01:07.732 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:08.203 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:08.203 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:07 smithi061 ceph-mon[127454]: Added host smithi149 2023-07-20T19:01:08.203 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:08.203 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:08.203 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:07 smithi061 ceph-mon[127454]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:09.002 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 -- 172.21.15.61:0/2896877574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 msgr2=0x7f7e5c105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:09.003 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 --2- 172.21.15.61:0/2896877574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c105560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009910 tx=0x7f7e58009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:09.003 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 -- 172.21.15.61:0/2896877574 shutdown_connections 2023-07-20T19:01:09.003 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 --2- 172.21.15.61:0/2896877574 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c105560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:09.003 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 -- 172.21.15.61:0/2896877574 >> 172.21.15.61:0/2896877574 conn(0x7f7e5c074c70 msgr2=0x7f7e5c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:09.003 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 -- 172.21.15.61:0/2896877574 shutdown_connections 2023-07-20T19:01:09.004 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 -- 172.21.15.61:0/2896877574 wait complete. 2023-07-20T19:01:09.005 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 Processor -- start 2023-07-20T19:01:09.005 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.950+0000 7f7e6429f700 1 -- start start 2023-07-20T19:01:09.005 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6429f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:09.005 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6429f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c1969e0 con 0x7f7e5c105180 2023-07-20T19:01:09.006 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6203b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:09.006 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6203b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c1964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57518/0 (socket says 172.21.15.61:57518) 2023-07-20T19:01:09.006 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6203b700 1 -- 172.21.15.61:0/1228685049 learned_addr learned my addr 172.21.15.61:0/1228685049 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:09.006 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6203b700 1 -- 172.21.15.61:0/1228685049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580095f0 con 0x7f7e5c105180 2023-07-20T19:01:09.006 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6203b700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c1964a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7e58006010 tx=0x7f7e58004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:09.006 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e5801d070 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e5800bbb0 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c196c40 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e58003930 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.951+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c1970c0 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.952+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f7e5800bd20 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.952+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c197300 con 0x7f7e5c105180 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.952+0000 7f7e537fe700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f7e480387b0 0x7f7e4803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:09.007 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.953+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7e5804c570 con 0x7f7e5c105180 2023-07-20T19:01:09.008 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.953+0000 7f7e6183a700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f7e480387b0 0x7f7e4803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:09.008 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.953+0000 7f7e6183a700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f7e480387b0 0x7f7e4803ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c006fd0 tx=0x7f7e4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:09.010 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:08.957+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7e58029360 con 0x7f7e5c105180 2023-07-20T19:01:09.211 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.159+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e5c068060 con 0x7f7e480387b0 2023-07-20T19:01:09.212 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.160+0000 7f7e537fe700 1 -- 172.21.15.61:0/1228685049 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f7e5c068060 con 0x7f7e480387b0 2023-07-20T19:01:09.213 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:01:09.213 INFO:teuthology.orchestra.run.smithi061.stdout:[{"addr": "172.21.15.61", "hostname": "smithi061", "labels": [], "status": ""}, {"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}] 2023-07-20T19:01:09.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:08 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:09.214 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f7e480387b0 msgr2=0x7f7e4803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f7e480387b0 0x7f7e4803ac70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c006fd0 tx=0x7f7e4c006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 msgr2=0x7f7e5c1964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c1964a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7e58006010 tx=0x7f7e58004ff0 comp rx=0 tx=0).stop 2023-07-20T19:01:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 shutdown_connections 2023-07-20T19:01:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f7e480387b0 0x7f7e4803ac70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 --2- 172.21.15.61:0/1228685049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e5c105180 0x7f7e5c1964a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:09.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 >> 172.21.15.61:0/1228685049 conn(0x7f7e5c074c70 msgr2=0x7f7e5c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:09.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 shutdown_connections 2023-07-20T19:01:09.216 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:09.162+0000 7f7e6429f700 1 -- 172.21.15.61:0/1228685049 wait complete. 2023-07-20T19:01:09.591 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2023-07-20T19:01:09.591 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T19:01:09.592 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2023-07-20T19:01:09.613 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T19:01:09.614 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:09.674 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2023-07-20T19:01:09.675 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch host add smithi156 2023-07-20T19:01:10.232 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:10.618 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:10 smithi061 ceph-mon[127454]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T19:01:10.618 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:10.618 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:10 smithi061 ceph-mon[127454]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.340+0000 7fa053fa9700 1 -- 172.21.15.61:0/2809548682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 msgr2=0x7fa04c105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.340+0000 7fa053fa9700 1 --2- 172.21.15.61:0/2809548682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c105560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa048009910 tx=0x7fa048009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 -- 172.21.15.61:0/2809548682 shutdown_connections 2023-07-20T19:01:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 --2- 172.21.15.61:0/2809548682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c105560 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:11.395 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 -- 172.21.15.61:0/2809548682 >> 172.21.15.61:0/2809548682 conn(0x7fa04c074c70 msgr2=0x7fa04c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:11.395 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 -- 172.21.15.61:0/2809548682 shutdown_connections 2023-07-20T19:01:11.395 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 -- 172.21.15.61:0/2809548682 wait complete. 2023-07-20T19:01:11.395 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 Processor -- start 2023-07-20T19:01:11.395 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.341+0000 7fa053fa9700 1 -- start start 2023-07-20T19:01:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa053fa9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa053fa9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c1969e0 con 0x7fa04c105180 2023-07-20T19:01:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa051d45700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa051d45700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c1964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57524/0 (socket says 172.21.15.61:57524) 2023-07-20T19:01:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa051d45700 1 -- 172.21.15.61:0/3332055005 learned_addr learned my addr 172.21.15.61:0/3332055005 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa051d45700 1 -- 172.21.15.61:0/3332055005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0480095f0 con 0x7fa04c105180 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa051d45700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c1964a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa048004f40 tx=0x7fa048005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa04801d070 con 0x7fa04c105180 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa048004370 con 0x7fa04c105180 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c196c40 con 0x7fa04c105180 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa048003970 con 0x7fa04c105180 2023-07-20T19:01:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.342+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c1970c0 con 0x7fa04c105180 2023-07-20T19:01:11.399 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.343+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa048003be0 con 0x7fa04c105180 2023-07-20T19:01:11.399 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.343+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c197300 con 0x7fa04c105180 2023-07-20T19:01:11.399 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.343+0000 7fa042ffd700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa038038800 0x7fa03803acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:11.399 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.343+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa04804c480 con 0x7fa04c105180 2023-07-20T19:01:11.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.344+0000 7fa051544700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa038038800 0x7fa03803acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:11.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.344+0000 7fa051544700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa038038800 0x7fa03803acc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa03c006fd0 tx=0x7fa03c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:11.401 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.348+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa04800f460 con 0x7fa04c105180 2023-07-20T19:01:11.602 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:11.549+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) v1 -- 0x7fa04c068060 con 0x7fa038038800 2023-07-20T19:01:12.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:11 smithi061 ceph-mon[127454]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:12.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:11 smithi061 ceph-mon[127454]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:13.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:13.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: Updating smithi149:/etc/ceph/ceph.conf 2023-07-20T19:01:13.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: Updating smithi149:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:13.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: Deploying cephadm binary to smithi156 2023-07-20T19:01:13.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:13.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:12 smithi061 ceph-mon[127454]: Updating smithi149:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T19:01:14.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:13 smithi061 ceph-mon[127454]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:14.406 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.353+0000 7fa042ffd700 1 -- 172.21.15.61:0/3332055005 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa04c068060 con 0x7fa038038800 2023-07-20T19:01:14.407 INFO:teuthology.orchestra.run.smithi061.stdout:Added host 'smithi156' with addr '172.21.15.156' 2023-07-20T19:01:14.408 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.355+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa038038800 msgr2=0x7fa03803acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.355+0000 7fa053fa9700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa038038800 0x7fa03803acc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa03c006fd0 tx=0x7fa03c006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 msgr2=0x7fa04c1964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c1964a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa048004f40 tx=0x7fa048005020 comp rx=0 tx=0).stop 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 shutdown_connections 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa038038800 0x7fa03803acc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 --2- 172.21.15.61:0/3332055005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa04c105180 0x7fa04c1964a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 >> 172.21.15.61:0/3332055005 conn(0x7fa04c074c70 msgr2=0x7fa04c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 shutdown_connections 2023-07-20T19:01:14.410 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:14.356+0000 7fa053fa9700 1 -- 172.21.15.61:0/3332055005 wait complete. 2023-07-20T19:01:14.799 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T19:01:15.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:15.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:15.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:15.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:15.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: Added host smithi156 2023-07-20T19:01:15.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:15.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:15.361 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.473+0000 7f72bf97d700 1 -- 172.21.15.61:0/1131979374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 msgr2=0x7f72b8105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.473+0000 7f72bf97d700 1 --2- 172.21.15.61:0/1131979374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b8105560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009910 tx=0x7f72b4009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.473+0000 7f72bf97d700 1 -- 172.21.15.61:0/1131979374 shutdown_connections 2023-07-20T19:01:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 --2- 172.21.15.61:0/1131979374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b8105560 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 -- 172.21.15.61:0/1131979374 >> 172.21.15.61:0/1131979374 conn(0x7f72b8074c70 msgr2=0x7f72b8075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 -- 172.21.15.61:0/1131979374 shutdown_connections 2023-07-20T19:01:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 -- 172.21.15.61:0/1131979374 wait complete. 2023-07-20T19:01:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 Processor -- start 2023-07-20T19:01:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 -- start start 2023-07-20T19:01:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b81956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.474+0000 7f72bf97d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8195bf0 con 0x7f72b8105180 2023-07-20T19:01:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bd719700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b81956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bd719700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b81956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57550/0 (socket says 172.21.15.61:57550) 2023-07-20T19:01:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bd719700 1 -- 172.21.15.61:0/2359293314 learned_addr learned my addr 172.21.15.61:0/2359293314 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bd719700 1 -- 172.21.15.61:0/2359293314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b40095f0 con 0x7f72b8105180 2023-07-20T19:01:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bd719700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b81956b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72b4006010 tx=0x7f72b4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72b401d070 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72b400bbb0 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72b4003930 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b8195e50 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.475+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b81962d0 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.476+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f72b400bd20 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.476+0000 7f72ae7fc700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f72a40387b0 0x7f72a403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.476+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b8061da0 con 0x7f72b8105180 2023-07-20T19:01:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.476+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f72b404c570 con 0x7f72b8105180 2023-07-20T19:01:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.476+0000 7f72bcf18700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f72a40387b0 0x7f72a403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.477+0000 7f72bcf18700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f72a40387b0 0x7f72a403ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72a8006fd0 tx=0x7f72a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:16.534 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.481+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f72b400b6f0 con 0x7f72b8105180 2023-07-20T19:01:16.714 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:16 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:16.714 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:16 smithi061 ceph-mon[127454]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:16.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.683+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72b8104450 con 0x7f72a40387b0 2023-07-20T19:01:16.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.684+0000 7f72ae7fc700 1 -- 172.21.15.61:0/2359293314 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f72b8104450 con 0x7f72a40387b0 2023-07-20T19:01:16.737 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:01:16.737 INFO:teuthology.orchestra.run.smithi061.stdout:[{"addr": "172.21.15.61", "hostname": "smithi061", "labels": [], "status": ""}, {"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2023-07-20T19:01:16.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f72a40387b0 msgr2=0x7f72a403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:16.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f72a40387b0 0x7f72a403ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72a8006fd0 tx=0x7f72a8006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:16.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 msgr2=0x7f72b81956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b81956b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72b4006010 tx=0x7f72b4004ff0 comp rx=0 tx=0).stop 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 shutdown_connections 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f72a40387b0 0x7f72a403ac70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 --2- 172.21.15.61:0/2359293314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b8105180 0x7f72b81956b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 >> 172.21.15.61:0/2359293314 conn(0x7f72b8074c70 msgr2=0x7f72b8102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 shutdown_connections 2023-07-20T19:01:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:16.686+0000 7f72bf97d700 1 -- 172.21.15.61:0/2359293314 wait complete. 2023-07-20T19:01:17.109 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T19:01:17.110 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T19:01:17.670 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:18.214 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:17 smithi061 ceph-mon[127454]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T19:01:18.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:18.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:17 smithi061 ceph-mon[127454]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:18.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 -- 172.21.15.61:0/1612009850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 msgr2=0x7f3e70105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:18.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 --2- 172.21.15.61:0/1612009850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70105560 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009910 tx=0x7f3e58009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:18.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 -- 172.21.15.61:0/1612009850 shutdown_connections 2023-07-20T19:01:18.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 --2- 172.21.15.61:0/1612009850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70105560 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:18.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 -- 172.21.15.61:0/1612009850 >> 172.21.15.61:0/1612009850 conn(0x7f3e70074c70 msgr2=0x7f3e70075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:18.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 -- 172.21.15.61:0/1612009850 shutdown_connections 2023-07-20T19:01:18.838 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.782+0000 7f3e763c5700 1 -- 172.21.15.61:0/1612009850 wait complete. 2023-07-20T19:01:18.838 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.783+0000 7f3e763c5700 1 Processor -- start 2023-07-20T19:01:18.838 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.783+0000 7f3e763c5700 1 -- start start 2023-07-20T19:01:18.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.783+0000 7f3e763c5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70194590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:18.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.783+0000 7f3e763c5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70194ad0 con 0x7f3e70105180 2023-07-20T19:01:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.783+0000 7f3e6ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70194590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70194590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57556/0 (socket says 172.21.15.61:57556) 2023-07-20T19:01:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6ffff700 1 -- 172.21.15.61:0/3868863442 learned_addr learned my addr 172.21.15.61:0/3868863442 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6ffff700 1 -- 172.21.15.61:0/3868863442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580095f0 con 0x7f3e70105180 2023-07-20T19:01:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6ffff700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70194590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e58006010 tx=0x7f3e58004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e5801d070 con 0x7f3e70105180 2023-07-20T19:01:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e70194d30 con 0x7f3e70105180 2023-07-20T19:01:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e5800bbb0 con 0x7f3e70105180 2023-07-20T19:01:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e58003930 con 0x7f3e70105180 2023-07-20T19:01:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.784+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e701951b0 con 0x7f3e70105180 2023-07-20T19:01:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.785+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3e58003b70 con 0x7f3e70105180 2023-07-20T19:01:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.785+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e70061da0 con 0x7f3e70105180 2023-07-20T19:01:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.785+0000 7f3e6d7fa700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3e5c03a9c0 0x7f3e5c03ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.785+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e5804c6e0 con 0x7f3e70105180 2023-07-20T19:01:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.785+0000 7f3e6f7fe700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3e5c03a9c0 0x7f3e5c03ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.786+0000 7f3e6f7fe700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3e5c03a9c0 0x7f3e5c03ce80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e60006fd0 tx=0x7f3e60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.790+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3e5800fd00 con 0x7f3e70105180 2023-07-20T19:01:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:18.997+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3e70103fb0 con 0x7f3e70105180 2023-07-20T19:01:19.505 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.451+0000 7f3e6d7fa700 1 -- 172.21.15.61:0/3868863442 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3e58026070 con 0x7f3e70105180 2023-07-20T19:01:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3e5c03a9c0 msgr2=0x7f3e5c03ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3e5c03a9c0 0x7f3e5c03ce80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e60006fd0 tx=0x7f3e60006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 msgr2=0x7f3e70194590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70194590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e58006010 tx=0x7f3e58004ff0 comp rx=0 tx=0).stop 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 shutdown_connections 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3e5c03a9c0 0x7f3e5c03ce80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 --2- 172.21.15.61:0/3868863442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e70105180 0x7f3e70194590 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 >> 172.21.15.61:0/3868863442 conn(0x7f3e70074c70 msgr2=0x7f3e700fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 shutdown_connections 2023-07-20T19:01:19.508 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:19.453+0000 7f3e763c5700 1 -- 172.21.15.61:0/3868863442 wait complete. 2023-07-20T19:01:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2023-07-20T19:01:19.820 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3868863442' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T19:01:19.820 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:19.820 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:19.820 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:19.820 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:19.821 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:01:19.821 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:19.821 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:19.822 INFO:tasks.cephadm:Adding mon.a on smithi061 2023-07-20T19:01:19.822 INFO:tasks.cephadm:Adding mon.b on smithi149 2023-07-20T19:01:19.822 INFO:tasks.cephadm:Adding mon.c on smithi156 2023-07-20T19:01:19.823 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c' 2023-07-20T19:01:20.408 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:20.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-20T19:01:20.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3868863442' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T19:01:20.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:20.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:20.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:20.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:20.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:21.614 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 -- 172.21.15.156:0/2995662105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 msgr2=0x7fd5fc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 --2- 172.21.15.156:0/2995662105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc104c50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009910 tx=0x7fd5ec009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 -- 172.21.15.156:0/2995662105 shutdown_connections 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 --2- 172.21.15.156:0/2995662105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc104c50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 -- 172.21.15.156:0/2995662105 >> 172.21.15.156:0/2995662105 conn(0x7fd5fc0757d0 msgr2=0x7fd5fc075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 -- 172.21.15.156:0/2995662105 shutdown_connections 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.560+0000 7fd604667700 1 -- 172.21.15.156:0/2995662105 wait complete. 2023-07-20T19:01:21.615 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.561+0000 7fd604667700 1 Processor -- start 2023-07-20T19:01:21.616 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.561+0000 7fd604667700 1 -- start start 2023-07-20T19:01:21.616 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.561+0000 7fd604667700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc100440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:21.616 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.561+0000 7fd604667700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1023c0 con 0x7fd5fc104870 2023-07-20T19:01:21.617 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd602403700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc100440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:21.617 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd602403700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc100440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.156:41844/0 (socket says 172.21.15.156:41844) 2023-07-20T19:01:21.617 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd602403700 1 -- 172.21.15.156:0/1555460881 learned_addr learned my addr 172.21.15.156:0/1555460881 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd602403700 1 -- 172.21.15.156:0/1555460881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0095f0 con 0x7fd5fc104870 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd602403700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc100440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec006010 tx=0x7fd5ec004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5ec01d070 con 0x7fd5fc104870 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc100980 con 0x7fd5fc104870 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5ec00bc10 con 0x7fd5fc104870 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5ec003910 con 0x7fd5fc104870 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.562+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc100e00 con 0x7fd5fc104870 2023-07-20T19:01:21.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.563+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd5ec00bd80 con 0x7fd5fc104870 2023-07-20T19:01:21.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.563+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc061da0 con 0x7fd5fc104870 2023-07-20T19:01:21.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.563+0000 7fd5f37fe700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fd5e8038790 0x7fd5e803ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:21.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.564+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd5ec04c610 con 0x7fd5fc104870 2023-07-20T19:01:21.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.564+0000 7fd601c02700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fd5e8038790 0x7fd5e803ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:21.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.564+0000 7fd601c02700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fd5e8038790 0x7fd5e803ac50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006fd0 tx=0x7fd5f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:21.622 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.569+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd5ec029360 con 0x7fd5fc104870 2023-07-20T19:01:21.814 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.761+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5fc1012d0 con 0x7fd5e8038790 2023-07-20T19:01:21.824 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.771+0000 7fd5f37fe700 1 -- 172.21.15.156:0/1555460881 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd5f8009e20 con 0x7fd5e8038790 2023-07-20T19:01:21.824 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mon update... 2023-07-20T19:01:21.826 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fd5e8038790 msgr2=0x7fd5e803ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:21.826 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fd5e8038790 0x7fd5e803ac50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006fd0 tx=0x7fd5f8006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:21.826 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 msgr2=0x7fd5fc100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:21.827 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc100440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec006010 tx=0x7fd5ec004ff0 comp rx=0 tx=0).stop 2023-07-20T19:01:21.827 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 shutdown_connections 2023-07-20T19:01:21.827 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fd5e8038790 0x7fd5e803ac50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:21.827 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 --2- 172.21.15.156:0/1555460881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5fc104870 0x7fd5fc100440 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:21.827 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 >> 172.21.15.156:0/1555460881 conn(0x7fd5fc0757d0 msgr2=0x7fd5fc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:21.827 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 shutdown_connections 2023-07-20T19:01:21.828 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:21.773+0000 7fd604667700 1 -- 172.21.15.156:0/1555460881 wait complete. 2023-07-20T19:01:21.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:21 smithi061 ceph-mon[127454]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:21 smithi061 ceph-mon[127454]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:21.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:21 smithi061 ceph-mon[127454]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T19:01:22.180 DEBUG:teuthology.orchestra.run.smithi149:mon.b> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T19:01:22.183 DEBUG:teuthology.orchestra.run.smithi156:mon.c> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T19:01:22.187 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T19:01:22.188 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T19:01:22.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:01:22.218 INFO:journalctl@ceph.mon.b.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:01:22.901 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:22.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:22.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:22.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:22.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:22.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:22.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:22.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:22.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:23.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:23 smithi061 ceph-mon[127454]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:23.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:23 smithi061 ceph-mon[127454]: Saving service mon spec with placement smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c;count:3 2023-07-20T19:01:23.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:23 smithi061 ceph-mon[127454]: Deploying daemon mon.c on smithi156 2023-07-20T19:01:24.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:24 smithi156 systemd[1]: Starting Ceph mon.c for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:01:24.945 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 -- 172.21.15.156:0/2524269716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 msgr2=0x7f645c106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:24.946 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 --2- 172.21.15.156:0/2524269716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c106670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f644c009910 tx=0x7f644c009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:24.946 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 -- 172.21.15.156:0/2524269716 shutdown_connections 2023-07-20T19:01:24.946 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 --2- 172.21.15.156:0/2524269716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c106670 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:24.946 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 -- 172.21.15.156:0/2524269716 >> 172.21.15.156:0/2524269716 conn(0x7f645c074c70 msgr2=0x7f645c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:24.946 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 -- 172.21.15.156:0/2524269716 shutdown_connections 2023-07-20T19:01:24.947 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.892+0000 7f646284f700 1 -- 172.21.15.156:0/2524269716 wait complete. 2023-07-20T19:01:24.947 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.893+0000 7f646284f700 1 Processor -- start 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.893+0000 7f646284f700 1 -- start start 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.893+0000 7f646284f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c195690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.893+0000 7f646284f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c195bd0 con 0x7f645c106290 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.894+0000 7f645bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c195690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.894+0000 7f645bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c195690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.156:41850/0 (socket says 172.21.15.156:41850) 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.894+0000 7f645bfff700 1 -- 172.21.15.156:0/3766013038 learned_addr learned my addr 172.21.15.156:0/3766013038 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:01:24.948 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.894+0000 7f645bfff700 1 -- 172.21.15.156:0/3766013038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0095f0 con 0x7f645c106290 2023-07-20T19:01:24.949 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.894+0000 7f645bfff700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c195690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f644c004f40 tx=0x7f644c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:24.949 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.894+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f644c01d070 con 0x7f645c106290 2023-07-20T19:01:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.895+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c195e30 con 0x7f645c106290 2023-07-20T19:01:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.895+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f644c004370 con 0x7f645c106290 2023-07-20T19:01:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.895+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f644c003970 con 0x7f645c106290 2023-07-20T19:01:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.895+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c1962b0 con 0x7f645c106290 2023-07-20T19:01:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.896+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f644c003c30 con 0x7f645c106290 2023-07-20T19:01:24.950 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.896+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c061da0 con 0x7f645c106290 2023-07-20T19:01:24.951 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.896+0000 7f64597fa700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f64440387b0 0x7f644403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:24.951 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.896+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f644c04c730 con 0x7f645c106290 2023-07-20T19:01:24.951 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.896+0000 7f645b7fe700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f64440387b0 0x7f644403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:24.951 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.897+0000 7f645b7fe700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f64440387b0 0x7f644403ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6450006fd0 tx=0x7f6450006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:24.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:24.901+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f644c029930 con 0x7f645c106290 2023-07-20T19:01:24.964 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:24 smithi061 ceph-mon[127454]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:25.234 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.180+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f645c1050d0 con 0x7f645c106290 2023-07-20T19:01:25.234 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.181+0000 7f64597fa700 1 -- 172.21.15.156:0/3766013038 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f644c026030 con 0x7f645c106290 2023-07-20T19:01:25.235 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T19:01:25.235 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","modified":"2023-07-20T18:59:45.096167Z","created":"2023-07-20T18:59:45.096167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T19:01:25.236 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f64440387b0 msgr2=0x7f644403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:25.237 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f64440387b0 0x7f644403ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6450006fd0 tx=0x7f6450006e40 comp rx=0 tx=0).stop 2023-07-20T19:01:25.237 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 msgr2=0x7f645c195690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:25.237 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c195690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f644c004f40 tx=0x7f644c005020 comp rx=0 tx=0).stop 2023-07-20T19:01:25.237 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 shutdown_connections 2023-07-20T19:01:25.237 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f64440387b0 0x7f644403ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:25.237 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 --2- 172.21.15.156:0/3766013038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f645c106290 0x7f645c195690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:25.238 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 >> 172.21.15.156:0/3766013038 conn(0x7f645c074c70 msgr2=0x7f645c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:25.238 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 shutdown_connections 2023-07-20T19:01:25.238 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:25.183+0000 7f646284f700 1 -- 172.21.15.156:0/3766013038 wait complete. 2023-07-20T19:01:25.238 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2023-07-20T19:01:25.370 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 podman[129166]: 2023-07-20T19:01:25.370 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T19:01:25.370 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T19:01:25.370 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: pidfile_write: ignore empty --pid-file 2023-07-20T19:01:25.371 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: load: jerasure load: lrc 2023-07-20T19:01:25.371 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: RocksDB version: 7.9.2 2023-07-20T19:01:25.371 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Git sha 0 2023-07-20T19:01:25.371 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T19:01:25.371 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: DB SUMMARY 2023-07-20T19:01:25.371 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: DB Session ID: KMO23BURDEDZ25ENEGBO 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: CURRENT file: CURRENT 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.error_if_exists: 0 2023-07-20T19:01:25.372 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.create_if_missing: 0 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.paranoid_checks: 1 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.env: 0x55a1f1381c60 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T19:01:25.373 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.info_log: 0x55a1f20cf2c0 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.statistics: (nil) 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.use_fsync: 0 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_log_file_size: 0 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T19:01:25.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.allow_fallocate: 1 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.use_direct_reads: 0 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T19:01:25.375 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.db_log_dir: 2023-07-20T19:01:25.376 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.wal_dir: 2023-07-20T19:01:25.376 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T19:01:25.376 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T19:01:25.376 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T19:01:25.376 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T19:01:25.376 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T19:01:25.377 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T19:01:25.377 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T19:01:25.377 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T19:01:25.377 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.write_buffer_manager: 0x55a1f1ff03c0 2023-07-20T19:01:25.377 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T19:01:25.377 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T19:01:25.378 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.unordered_write: 0 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.row_cache: None 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.wal_filter: None 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 bash[129166]: 92d6cfc459bd1ae0e6bd8fab3549fd177a0225a7e4b070473920b5f59e8e6427 2023-07-20T19:01:25.379 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T19:01:25.380 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T19:01:25.380 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.two_write_queues: 0 2023-07-20T19:01:25.380 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T19:01:25.380 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.wal_compression: 0 2023-07-20T19:01:25.380 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.atomic_flush: 0 2023-07-20T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.log_readahead_size: 0 2023-07-20T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T19:01:25.382 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T19:01:25.382 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T19:01:25.382 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T19:01:25.384 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T19:01:25.385 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T19:01:25.385 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T19:01:25.385 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_background_jobs: 2 2023-07-20T19:01:25.386 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_background_compactions: -1 2023-07-20T19:01:25.386 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_subcompactions: 1 2023-07-20T19:01:25.386 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T19:01:25.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T19:01:25.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T19:01:25.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T19:01:25.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T19:01:25.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T19:01:25.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T19:01:25.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T19:01:25.389 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_open_files: -1 2023-07-20T19:01:25.389 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T19:01:25.389 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T19:01:25.389 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T19:01:25.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T19:01:25.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_background_flushes: -1 2023-07-20T19:01:25.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Compression algorithms supported: 2023-07-20T19:01:25.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T19:01:25.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kZSTD supported: 0 2023-07-20T19:01:25.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kXpressCompression supported: 0 2023-07-20T19:01:25.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T19:01:25.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kZlibCompression supported: 1 2023-07-20T19:01:25.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kSnappyCompression supported: 1 2023-07-20T19:01:25.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kLZ4Compression supported: 1 2023-07-20T19:01:25.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: kBZip2Compression supported: 0 2023-07-20T19:01:25.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T19:01:25.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T19:01:25.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.merge_operator: 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_filter: None 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T19:01:25.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a1f20cf3c0) 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks: 1 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_top_level_index_and_filter: 1 2023-07-20T19:01:25.394 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_type: 0 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_index_type: 0 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_shortening: 1 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: checksum: 4 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: no_block_cache: 0 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache: 0x55a1f208f350 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_name: BinnedLRUCache 2023-07-20T19:01:25.395 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_options: 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: capacity : 536870912 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_shard_bits : 4 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: strict_capacity_limit : 0 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: high_pri_pool_ratio: 0.000 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_compressed: (nil) 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: persistent_cache: (nil) 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size: 4096 2023-07-20T19:01:25.396 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size_deviation: 10 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_restart_interval: 16 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_block_restart_interval: 1 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: metadata_block_size: 4096 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: partition_filters: 0 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: use_delta_encoding: 1 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: filter_policy: bloomfilter 2023-07-20T19:01:25.397 INFO:journalctl@ceph.mon.c.smithi156.stdout: whole_key_filtering: 1 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: verify_compression: 0 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: read_amp_bytes_per_bit: 0 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: format_version: 5 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_index_compression: 1 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_align: 0 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: max_auto_readahead_size: 262144 2023-07-20T19:01:25.398 INFO:journalctl@ceph.mon.c.smithi156.stdout: prepopulate_block_cache: 0 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout: initial_auto_readahead_size: 8192 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression: NoCompression 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T19:01:25.399 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T19:01:25.400 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.num_levels: 7 2023-07-20T19:01:25.400 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T19:01:25.400 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T19:01:25.400 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T19:01:25.400 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T19:01:25.400 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T19:01:25.401 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T19:01:25.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T19:01:25.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T19:01:25.404 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T19:01:25.405 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T19:01:25.406 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.table_properties_collectors: 2023-07-20T19:01:25.407 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.inplace_update_support: 0 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.bloom_locality: 0 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.max_successive_merges: 0 2023-07-20T19:01:25.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.ttl: 2592000 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T19:01:25.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.enable_blob_files: false 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.min_blob_size: 0 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T19:01:25.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ff4600bf-007e-4360-ac95-cdbd65109886 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879685251150, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T19:01:25.411 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T19:01:25.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879685251742, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689879685, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff4600bf-007e-4360-ac95-cdbd65109886", "db_session_id": "KMO23BURDEDZ25ENEGBO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T19:01:25.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879685251837, "job": 1, "event": "recovery_finished"} 2023-07-20T19:01:25.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T19:01:25.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T19:01:25.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a1f2128000 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: DB pointer 0x55a1f2114000 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** DB Stats ** 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T19:01:25.413 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T19:01:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T19:01:25.415 INFO:journalctl@ceph.mon.c.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T19:01:25.416 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache BinnedLRUCache@0x55a1f208f350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T19:01:25.417 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: starting mon.c rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(???) e0 preinit fsid 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).mds e1 new map 2023-07-20T19:01:25.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout: e1 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout: legacy client fscid: -1 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout: No filesystems configured 2023-07-20T19:01:25.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T19:01:25.420 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Added host smithi156 2023-07-20T19:01:25.421 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:25.422 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3868863442' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T19:01:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:01:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:25.424 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:25.424 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-20T19:01:25.424 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3868863442' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T19:01:25.424 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:25.425 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:25.425 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.425 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.425 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.426 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:25.426 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:25.426 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T19:01:25.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:25.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:25.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:25.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:25.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:25.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:25.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:25.429 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:25.429 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Saving service mon spec with placement smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c;count:3 2023-07-20T19:01:25.429 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: Deploying daemon mon.c on smithi156 2023-07-20T19:01:25.430 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:25.430 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T19:01:25.430 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-20T19:01:25.430 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expand map: {default=false} 2023-07-20T19:01:25.431 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta from 'false' to 'false' 2023-07-20T19:01:25.431 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expanded map: {default=false} 2023-07-20T19:01:25.431 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expand map: {default=info} 2023-07-20T19:01:25.431 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta from 'info' to 'info' 2023-07-20T19:01:25.432 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expanded map: {default=info} 2023-07-20T19:01:25.432 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expand map: {default=daemon} 2023-07-20T19:01:25.432 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T19:01:25.433 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T19:01:25.433 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expand map: {default=debug} 2023-07-20T19:01:25.433 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T19:01:25.433 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 ceph-mon[129228]: expand_channel_meta expanded map: {default=debug} 2023-07-20T19:01:25.433 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:25 smithi156 systemd[1]: Started Ceph mon.c for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:01:26.764 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T19:01:26.765 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T19:01:27.332 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:01:30.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:01:30.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: mon.a calling monitor election 2023-07-20T19:01:30.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:30.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: mon.c calling monitor election 2023-07-20T19:01:30.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:30.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: monmap e2: 2 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: fsmap 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: mgrmap e13: x(active, since 42s) 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: overall HEALTH_OK 2023-07-20T19:01:30.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:30.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:30.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:30.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:30.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:30.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:01:30.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: mon.a calling monitor election 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: mon.c calling monitor election 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: monmap e2: 2 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: fsmap 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:30.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: mgrmap e13: x(active, since 42s) 2023-07-20T19:01:30.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: overall HEALTH_OK 2023-07-20T19:01:30.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:30.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:30.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:30.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:30.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 -- 172.21.15.156:0/3186276426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 msgr2=0x7fe2e4007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/3186276426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2e4007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8076160 tx=0x7fe2e8009e80 comp rx=0 tx=0).stop 2023-07-20T19:01:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 -- 172.21.15.156:0/3186276426 shutdown_connections 2023-07-20T19:01:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/3186276426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2e4007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 -- 172.21.15.156:0/3186276426 >> 172.21.15.156:0/3186276426 conn(0x7fe2f8074d20 msgr2=0x7fe2f8075130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 -- 172.21.15.156:0/3186276426 shutdown_connections 2023-07-20T19:01:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.579+0000 7fe2fc94e700 1 -- 172.21.15.156:0/3186276426 wait complete. 2023-07-20T19:01:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2fc94e700 1 Processor -- start 2023-07-20T19:01:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2fc94e700 1 -- start start 2023-07-20T19:01:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2fc94e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2f819d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2fc94e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:31.636 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2fc94e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819e260 con 0x7fe2e40057d0 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2fc94e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819e3d0 con 0x7fe2f819dd20 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48316/0 (socket says 172.21.15.156:48316) 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 -- 172.21.15.156:0/2059347281 learned_addr learned my addr 172.21.15.156:0/2059347281 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f659c700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2f819d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 -- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 msgr2=0x7fe2f8197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T19:01:31.637 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 -- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 msgr2=0x7fe2f8197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.581+0000 7fe2f5d9b700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2f659c700 1 -- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 msgr2=0x7fe2f8197860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2f659c700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2f659c700 1 -- 172.21.15.156:0/2059347281 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e80098d0 con 0x7fe2e40057d0 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2f659c700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2f819d7e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8076160 tx=0x7fe2e8017820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe2e8003e30 con 0x7fe2e40057d0 2023-07-20T19:01:31.638 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f8197e60 con 0x7fe2e40057d0 2023-07-20T19:01:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2e8005570 con 0x7fe2e40057d0 2023-07-20T19:01:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe2e802aa70 con 0x7fe2e40057d0 2023-07-20T19:01:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.582+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f81983b0 con 0x7fe2e40057d0 2023-07-20T19:01:31.639 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.583+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fe2e8027070 con 0x7fe2e40057d0 2023-07-20T19:01:31.640 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.583+0000 7fe2ef7fe700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe2d80385f0 0x7fe2d803aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:31.640 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.583+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f8061da0 con 0x7fe2e40057d0 2023-07-20T19:01:31.640 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.583+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe2e804e9d0 con 0x7fe2e40057d0 2023-07-20T19:01:31.640 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.583+0000 7fe2f5d9b700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe2d80385f0 0x7fe2d803aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:31.640 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.584+0000 7fe2f5d9b700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe2d80385f0 0x7fe2d803aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009180 tx=0x7fe2e0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:31.642 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.589+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe2e8020450 con 0x7fe2e40057d0 2023-07-20T19:01:31.714 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:31 smithi061 ceph-mon[127454]: Deploying daemon mon.b on smithi149 2023-07-20T19:01:31.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:31 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:31.906 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.853+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2f8198670 con 0x7fe2e40057d0 2023-07-20T19:01:31.907 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.854+0000 7fe2ef7fe700 1 -- 172.21.15.156:0/2059347281 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fe2e801e360 con 0x7fe2e40057d0 2023-07-20T19:01:31.907 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T19:01:31.908 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","modified":"2023-07-20T19:01:25.277316Z","created":"2023-07-20T18:59:45.096167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T19:01:31.909 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.855+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe2d80385f0 msgr2=0x7fe2d803aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.855+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe2d80385f0 0x7fe2d803aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009180 tx=0x7fe2e0008040 comp rx=0 tx=0).stop 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.855+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 msgr2=0x7fe2f819d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2f819d7e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8076160 tx=0x7fe2e8017820 comp rx=0 tx=0).stop 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 shutdown_connections 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe2d80385f0 0x7fe2d803aab0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe2e40057d0 0x7fe2f819d7e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 --2- 172.21.15.156:0/2059347281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2f819dd20 0x7fe2f8197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:31.910 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 >> 172.21.15.156:0/2059347281 conn(0x7fe2f8074d20 msgr2=0x7fe2f80fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 shutdown_connections 2023-07-20T19:01:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:31.856+0000 7fe2fc94e700 1 -- 172.21.15.156:0/2059347281 wait complete. 2023-07-20T19:01:31.911 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2023-07-20T19:01:31.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:31 smithi156 ceph-mon[129228]: Deploying daemon mon.b on smithi149 2023-07-20T19:01:31.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:31 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:32.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:32 smithi156 ceph-mon[129228]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:32.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:32 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/2059347281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T19:01:32.714 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:32 smithi061 ceph-mon[127454]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:32.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:32 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/2059347281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T19:01:33.346 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:32 smithi149 systemd[1]: Starting Ceph mon.b for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:01:33.399 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T19:01:33.400 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T19:01:33.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 podman[128845]: 2023-07-20T19:01:33.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T19:01:33.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T19:01:33.598 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pidfile_write: ignore empty --pid-file 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: load: jerasure load: lrc 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: RocksDB version: 7.9.2 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Git sha 0 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: DB SUMMARY 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: DB Session ID: 4L5B5FU2XEZZPCFCX9XB 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: CURRENT file: CURRENT 2023-07-20T19:01:33.599 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.error_if_exists: 0 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.create_if_missing: 0 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.paranoid_checks: 1 2023-07-20T19:01:33.600 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.env: 0x55a7331f4c60 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.info_log: 0x55a7359532c0 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.statistics: (nil) 2023-07-20T19:01:33.601 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.use_fsync: 0 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_log_file_size: 0 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.allow_fallocate: 1 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T19:01:33.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.use_direct_reads: 0 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.db_log_dir: 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.wal_dir: 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T19:01:33.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.write_buffer_manager: 0x55a7358743c0 2023-07-20T19:01:33.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T19:01:33.605 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.unordered_write: 0 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.row_cache: None 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.wal_filter: None 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T19:01:33.606 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.two_write_queues: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.wal_compression: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.atomic_flush: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T19:01:33.607 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.log_readahead_size: 0 2023-07-20T19:01:33.608 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T19:01:33.608 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T19:01:33.608 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T19:01:33.608 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T19:01:33.608 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T19:01:33.608 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T19:01:33.609 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T19:01:33.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_background_jobs: 2 2023-07-20T19:01:33.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_background_compactions: -1 2023-07-20T19:01:33.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_subcompactions: 1 2023-07-20T19:01:33.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T19:01:33.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T19:01:33.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_open_files: -1 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T19:01:33.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_background_flushes: -1 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Compression algorithms supported: 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kZSTD supported: 0 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kXpressCompression supported: 0 2023-07-20T19:01:33.612 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kZlibCompression supported: 1 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kSnappyCompression supported: 1 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kLZ4Compression supported: 1 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: kBZip2Compression supported: 0 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T19:01:33.613 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.merge_operator: 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_filter: None 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T19:01:33.614 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a7359533c0) 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks: 1 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_top_level_index_and_filter: 1 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_type: 0 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_index_type: 0 2023-07-20T19:01:33.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_shortening: 1 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: checksum: 4 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: no_block_cache: 0 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache: 0x55a7359131f0 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_name: BinnedLRUCache 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_options: 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: capacity : 536870912 2023-07-20T19:01:33.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_shard_bits : 4 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: strict_capacity_limit : 0 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: high_pri_pool_ratio: 0.000 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_compressed: (nil) 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: persistent_cache: (nil) 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size: 4096 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size_deviation: 10 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_restart_interval: 16 2023-07-20T19:01:33.617 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_block_restart_interval: 1 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: metadata_block_size: 4096 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: partition_filters: 0 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_delta_encoding: 1 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: filter_policy: bloomfilter 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: whole_key_filtering: 1 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: verify_compression: 0 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: read_amp_bytes_per_bit: 0 2023-07-20T19:01:33.618 INFO:journalctl@ceph.mon.b.smithi149.stdout: format_version: 5 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_index_compression: 1 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_align: 0 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout: max_auto_readahead_size: 262144 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout: prepopulate_block_cache: 0 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout: initial_auto_readahead_size: 8192 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T19:01:33.619 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression: NoCompression 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.num_levels: 7 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T19:01:33.620 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 bash[128845]: 8e9e3aa0002bae5ed0d1652318d99ade3c4f2033e5616a688efc88eb0cc0a1f8 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T19:01:33.621 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T19:01:33.622 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T19:01:33.623 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T19:01:33.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T19:01:33.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T19:01:33.626 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.table_properties_collectors: 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.inplace_update_support: 0 2023-07-20T19:01:33.627 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.bloom_locality: 0 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.max_successive_merges: 0 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T19:01:33.628 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.ttl: 2592000 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.enable_blob_files: false 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.min_blob_size: 0 2023-07-20T19:01:33.629 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T19:01:33.630 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7ff15dbd-c64f-4de9-97a4-c910c072485d 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879693514005, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879693514438, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689879693, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7ff15dbd-c64f-4de9-97a4-c910c072485d", "db_session_id": "4L5B5FU2XEZZPCFCX9XB", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689879693514507, "job": 1, "event": "recovery_finished"} 2023-07-20T19:01:33.631 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a7359ac000 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: DB pointer 0x55a735998000 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** DB Stats ** 2023-07-20T19:01:33.632 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2023-07-20T19:01:33.633 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2023-07-20T19:01:33.634 INFO:journalctl@ceph.mon.b.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T19:01:33.635 INFO:journalctl@ceph.mon.b.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T19:01:33.636 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache BinnedLRUCache@0x55a7359131f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: using public_addr v2:172.21.15.149:0/0 -> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: starting mon.b rank -1 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(???) e0 preinit fsid 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:01:33.637 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout: e1 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout: legacy client fscid: -1 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout: No filesystems configured 2023-07-20T19:01:33.638 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T19:01:33.639 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Added host smithi156 2023-07-20T19:01:33.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3868863442' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T19:01:33.641 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-20T19:01:33.642 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3868863442' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:33.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:33.644 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Saving service mon spec with placement smithi061:172.21.15.61=a;smithi149:172.21.15.149=b;smithi156:172.21.15.156=c;count:3 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Deploying daemon mon.c on smithi156 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.a calling monitor election 2023-07-20T19:01:33.645 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.c calling monitor election 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.646 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: monmap e2: 2 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: fsmap 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mgrmap e13: x(active, since 42s) 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: overall HEALTH_OK 2023-07-20T19:01:33.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: Deploying daemon mon.b on smithi149 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:33.648 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/2059347281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expand map: {default=false} 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta from 'false' to 'false' 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expanded map: {default=false} 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expand map: {default=info} 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta from 'info' to 'info' 2023-07-20T19:01:33.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expanded map: {default=info} 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expand map: {default=daemon} 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expand map: {default=debug} 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: expand_channel_meta expanded map: {default=debug} 2023-07-20T19:01:33.650 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 ceph-mon[128885]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T19:01:33.965 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:01:34.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:33 smithi149 systemd[1]: Started Ceph mon.b for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:01:38.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 -- 172.21.15.156:0/1431979609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 msgr2=0x7fa100007820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:38.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/1431979609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa100007820 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa108005fa0 tx=0x7fa10800f4a0 comp rx=0 tx=0).stop 2023-07-20T19:01:38.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 -- 172.21.15.156:0/1431979609 shutdown_connections 2023-07-20T19:01:38.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/1431979609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa100007820 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:38.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 -- 172.21.15.156:0/1431979609 >> 172.21.15.156:0/1431979609 conn(0x7fa1180744c0 msgr2=0x7fa1180748d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:38.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 -- 172.21.15.156:0/1431979609 shutdown_connections 2023-07-20T19:01:38.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.709+0000 7fa11dcf9700 1 -- 172.21.15.156:0/1431979609 wait complete. 2023-07-20T19:01:38.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.710+0000 7fa11dcf9700 1 Processor -- start 2023-07-20T19:01:38.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.710+0000 7fa11dcf9700 1 -- start start 2023-07-20T19:01:38.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.710+0000 7fa11dcf9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa118193610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:38.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.710+0000 7fa11dcf9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:38.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.710+0000 7fa11dcf9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 0x7fa1181a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:38.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa11dcf9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181a89a0 con 0x7fa1180ffd90 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa11dcf9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181a8b10 con 0x7fa1181941f0 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa11dcf9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181a8cd0 con 0x7fa118197560 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48338/0 (socket says 172.21.15.156:48338) 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 -- 172.21.15.156:0/2318331822 learned_addr learned my addr 172.21.15.156:0/2318331822 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa118193610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:38.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 msgr2=0x7fa118193b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 msgr2=0x7fa118193b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa117fff700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 0x7fa1181a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa116ffd700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa117fff700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 msgr2=0x7fa1181a81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa117fff700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 msgr2=0x7fa1181a81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa117fff700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 0x7fa1181a81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T19:01:38.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 msgr2=0x7fa118193b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa117fff700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 0x7fa1181a81d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 msgr2=0x7fa1181a81d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 0x7fa1181a81d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 -- 172.21.15.156:0/2318331822 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1080149d0 con 0x7fa1180ffd90 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.711+0000 7fa1177fe700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa118193610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa10801d000 tx=0x7fa108005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.712+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108019070 con 0x7fa1180ffd90 2023-07-20T19:01:38.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.712+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa10801a040 con 0x7fa1180ffd90 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.712+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1181a8f50 con 0x7fa1180ffd90 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.712+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10801ece0 con 0x7fa1180ffd90 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.712+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1181a9470 con 0x7fa1180ffd90 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.713+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa108028440 con 0x7fa1180ffd90 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.713+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11818dbf0 con 0x7fa1180ffd90 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.713+0000 7fa114ff9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa104038c80 0x7fa10403b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:38.771 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.713+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa10804d5a0 con 0x7fa1180ffd90 2023-07-20T19:01:38.772 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.713+0000 7fa116ffd700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa104038c80 0x7fa10403b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:38.772 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.714+0000 7fa116ffd700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa104038c80 0x7fa10403b140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa118194d60 tx=0x7fa10c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:38.772 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:38.718+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa10801ee40 con 0x7fa1180ffd90 2023-07-20T19:01:38.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:01:38.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: mon.c calling monitor election 2023-07-20T19:01:38.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: mon.a calling monitor election 2023-07-20T19:01:38.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T19:01:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: monmap e3: 3 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: fsmap 2023-07-20T19:01:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: mgrmap e13: x(active, since 50s) 2023-07-20T19:01:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T19:01:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T19:01:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T19:01:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2023-07-20T19:01:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:39.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.005+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa118194aa0 con 0x7fa1180ffd90 2023-07-20T19:01:39.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.006+0000 7fa114ff9700 1 -- 172.21.15.156:0/2318331822 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7fa10802f340 con 0x7fa1180ffd90 2023-07-20T19:01:39.061 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-20T19:01:39.061 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","modified":"2023-07-20T19:01:33.547822Z","created":"2023-07-20T18:59:45.096167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T19:01:39.062 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.008+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa104038c80 msgr2=0x7fa10403b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:39.062 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.008+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa104038c80 0x7fa10403b140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa118194d60 tx=0x7fa10c008040 comp rx=0 tx=0).stop 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.008+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 msgr2=0x7fa118193610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.008+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa118193610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa10801d000 tx=0x7fa108005390 comp rx=0 tx=0).stop 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 shutdown_connections 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa104038c80 0x7fa10403b140 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa1181941f0 0x7fa1181a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1180ffd90 0x7fa118193610 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:39.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 --2- 172.21.15.156:0/2318331822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa118197560 0x7fa118193b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:39.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 >> 172.21.15.156:0/2318331822 conn(0x7fa1180744c0 msgr2=0x7fa118102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:39.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 shutdown_connections 2023-07-20T19:01:39.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:39.009+0000 7fa11dcf9700 1 -- 172.21.15.156:0/2318331822 wait complete. 2023-07-20T19:01:39.064 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 3 2023-07-20T19:01:39.084 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:01:39.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: mon.c calling monitor election 2023-07-20T19:01:39.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: mon.a calling monitor election 2023-07-20T19:01:39.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:39.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:39.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:39.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:39.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: monmap e3: 3 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: fsmap 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: mgrmap e13: x(active, since 50s) 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T19:01:39.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T19:01:39.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T19:01:39.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2023-07-20T19:01:39.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:39.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:39.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:39.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:39.415 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T19:01:39.416 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T19:01:39.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:39.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:39 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/2318331822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T19:01:39.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:39.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:39 smithi061 ceph-mon[127454]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:39.993 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:40.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:40.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:39 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/2318331822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T19:01:40.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:40.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:39 smithi156 ceph-mon[129228]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:41.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:01:41.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: mon.c calling monitor election 2023-07-20T19:01:41.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: mon.a calling monitor election 2023-07-20T19:01:41.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:01:41.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T19:01:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: monmap e3: 3 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: fsmap 2023-07-20T19:01:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:41.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: mgrmap e13: x(active, since 50s) 2023-07-20T19:01:41.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T19:01:41.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T19:01:41.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T19:01:41.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2023-07-20T19:01:41.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:41.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/2318331822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T19:01:41.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:41.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:40 smithi149 ceph-mon[128885]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:41.201 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 -- 172.21.15.61:0/2237920623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6708105180 msgr2=0x7f6708105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 --2- 172.21.15.61:0/2237920623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6708105180 0x7f6708105560 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009910 tx=0x7f66f8009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 -- 172.21.15.61:0/2237920623 shutdown_connections 2023-07-20T19:01:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 --2- 172.21.15.61:0/2237920623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6708105180 0x7f6708105560 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 -- 172.21.15.61:0/2237920623 >> 172.21.15.61:0/2237920623 conn(0x7f6708074c70 msgr2=0x7f6708075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 -- 172.21.15.61:0/2237920623 shutdown_connections 2023-07-20T19:01:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.147+0000 7f670d9d0700 1 -- 172.21.15.61:0/2237920623 wait complete. 2023-07-20T19:01:41.204 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 Processor -- start 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 -- start start 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6708105180 0x7f6708196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6708196bd0 0x7f6708197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 0x7f67081a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67081a2270 con 0x7f670819f700 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67081a23e0 con 0x7f6708105180 2023-07-20T19:01:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f670d9d0700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67081a25a0 con 0x7f6708196bd0 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f67077fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 0x7f67081a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.148+0000 7f67077fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 0x7f67081a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57566/0 (socket says 172.21.15.61:57566) 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 -- 172.21.15.61:0/3935846329 learned_addr learned my addr 172.21.15.61:0/3935846329 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f6706ffd700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6708105180 0x7f6708196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67067fc700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6708196bd0 0x7f6708197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 -- 172.21.15.61:0/3935846329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6708196bd0 msgr2=0x7f6708197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6708196bd0 0x7f6708197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 -- 172.21.15.61:0/3935846329 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6708105180 msgr2=0x7f6708196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6708105180 0x7f6708196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 -- 172.21.15.61:0/3935846329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f80095f0 con 0x7f670819f700 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67067fc700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6708196bd0 0x7f6708197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f67077fe700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 0x7f67081a1af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f670000ba50 tx=0x7f670000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670000c9b0 con 0x7f670819f700 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6700014440 con 0x7f670819f700 2023-07-20T19:01:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67081a2880 con 0x7f670819f700 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700007590 con 0x7f670819f700 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.149+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67081a2d80 con 0x7f670819f700 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.150+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f67000077b0 con 0x7f670819f700 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.150+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708061da0 con 0x7f670819f700 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.150+0000 7f670c9ce700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f66f0038c30 0x7f66f003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.150+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f670004aab0 con 0x7f670819f700 2023-07-20T19:01:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.150+0000 7f6706ffd700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f66f0038c30 0x7f66f003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:41.209 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.151+0000 7f6706ffd700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f66f0038c30 0x7f66f003b0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f8006010 tx=0x7f66f800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:41.209 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.155+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f670001b8b0 con 0x7f670819f700 2023-07-20T19:01:41.428 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.373+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6708067f80 con 0x7f670819f700 2023-07-20T19:01:41.428 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.374+0000 7f670c9ce700 1 -- 172.21.15.61:0/3935846329 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f6700012340 con 0x7f670819f700 2023-07-20T19:01:41.429 INFO:teuthology.orchestra.run.smithi061.stdout:# minimal ceph.conf for 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:01:41.429 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2023-07-20T19:01:41.429 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:01:41.429 INFO:teuthology.orchestra.run.smithi061.stdout: mon_host = [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-07-20T19:01:41.431 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f66f0038c30 msgr2=0x7f66f003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:41.431 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f66f0038c30 0x7f66f003b0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f8006010 tx=0x7f66f800b540 comp rx=0 tx=0).stop 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 msgr2=0x7f67081a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 0x7f67081a1af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f670000ba50 tx=0x7f670000bd60 comp rx=0 tx=0).stop 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 shutdown_connections 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f66f0038c30 0x7f66f003b0f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f670819f700 0x7f67081a1af0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6708105180 0x7f6708196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.432 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 --2- 172.21.15.61:0/3935846329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6708196bd0 0x7f6708197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:41.433 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 >> 172.21.15.61:0/3935846329 conn(0x7f6708074c70 msgr2=0x7f67080fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:41.433 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 shutdown_connections 2023-07-20T19:01:41.433 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:41.377+0000 7f670d9d0700 1 -- 172.21.15.61:0/3935846329 wait complete. 2023-07-20T19:01:41.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: mon.b calling monitor election 2023-07-20T19:01:41.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Updating smithi061:/etc/ceph/ceph.conf 2023-07-20T19:01:41.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Updating smithi149:/etc/ceph/ceph.conf 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: mon.b calling monitor election 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: mon.a calling monitor election 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: mon.c calling monitor election 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: monmap e3: 3 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:41.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: fsmap 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: mgrmap e13: x(active, since 52s) 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Cluster is now healthy 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: overall HEALTH_OK 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Updating smithi149:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: Updating smithi061:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:41.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:41.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:41.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: mon.b calling monitor election 2023-07-20T19:01:41.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Updating smithi061:/etc/ceph/ceph.conf 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Updating smithi149:/etc/ceph/ceph.conf 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: mon.b calling monitor election 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: mon.a calling monitor election 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: mon.c calling monitor election 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T19:01:41.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: monmap e3: 3 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: fsmap 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: mgrmap e13: x(active, since 52s) 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Cluster is now healthy 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: overall HEALTH_OK 2023-07-20T19:01:41.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Updating smithi149:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: Updating smithi061:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:41.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:41.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:41.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: mon.b calling monitor election 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Updating smithi061:/etc/ceph/ceph.conf 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Updating smithi149:/etc/ceph/ceph.conf 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: mon.b calling monitor election 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: mon.a calling monitor election 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: mon.c calling monitor election 2023-07-20T19:01:41.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: monmap e3: 3 mons at {a=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: fsmap 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: mgrmap e13: x(active, since 52s) 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Cluster is now healthy 2023-07-20T19:01:41.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: overall HEALTH_OK 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Updating smithi149:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Updating smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: Updating smithi061:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:41.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:41.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:41.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:41.827 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T19:01:41.828 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:01:41.828 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T19:01:41.860 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:01:41.860 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:41.930 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T19:01:41.930 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T19:01:41.963 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T19:01:41.963 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:42.031 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T19:01:42.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T19:01:42.069 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T19:01:42.069 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T19:01:42.140 INFO:tasks.cephadm:Adding mgr.x on smithi061 2023-07-20T19:01:42.141 INFO:tasks.cephadm:Adding mgr.y on smithi149 2023-07-20T19:01:42.141 INFO:tasks.cephadm:Adding mgr.z on smithi156 2023-07-20T19:01:42.141 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi061=x;smithi149=y;smithi156=z' 2023-07-20T19:01:42.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:42 smithi156 ceph-mon[129228]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T19:01:42.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:42 smithi156 ceph-mon[129228]: Reconfiguring daemon mon.a on smithi061 2023-07-20T19:01:42.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:42 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3935846329' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:42.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:42.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:42 smithi156 ceph-mon[129228]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:42.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:42 smithi061 ceph-mon[127454]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T19:01:42.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:42 smithi061 ceph-mon[127454]: Reconfiguring daemon mon.a on smithi061 2023-07-20T19:01:42.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:42 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3935846329' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:42.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:42.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:42 smithi061 ceph-mon[127454]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:42.729 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:01:42.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:42 smithi149 ceph-mon[128885]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T19:01:42.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:42 smithi149 ceph-mon[128885]: Reconfiguring daemon mon.a on smithi061 2023-07-20T19:01:42.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:42 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3935846329' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:42.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:01:42.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:42 smithi149 ceph-mon[128885]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:43.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: Reconfiguring mon.b (monmap changed)... 2023-07-20T19:01:43.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:43.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:43.514 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:43.514 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: Reconfiguring daemon mon.b on smithi149 2023-07-20T19:01:43.514 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: Reconfiguring mon.b (monmap changed)... 2023-07-20T19:01:43.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:43.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:43.617 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:43.618 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: Reconfiguring daemon mon.b on smithi149 2023-07-20T19:01:43.618 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: Reconfiguring mon.b (monmap changed)... 2023-07-20T19:01:43.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:43.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:43.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:43.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: Reconfiguring daemon mon.b on smithi149 2023-07-20T19:01:43.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:43.970 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 -- 172.21.15.156:0/3236541926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca200ff290 msgr2=0x7fca200ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:43.970 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 --2- 172.21.15.156:0/3236541926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca200ff290 0x7fca200ff6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009910 tx=0x7fca1c009c20 comp rx=0 tx=0).stop 2023-07-20T19:01:43.970 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 -- 172.21.15.156:0/3236541926 shutdown_connections 2023-07-20T19:01:43.971 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 --2- 172.21.15.156:0/3236541926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca200ff290 0x7fca200ff6b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:43.971 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 -- 172.21.15.156:0/3236541926 >> 172.21.15.156:0/3236541926 conn(0x7fca200fa810 msgr2=0x7fca200fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:43.971 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 -- 172.21.15.156:0/3236541926 shutdown_connections 2023-07-20T19:01:43.971 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.916+0000 7fca2788d700 1 -- 172.21.15.156:0/3236541926 wait complete. 2023-07-20T19:01:43.973 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 Processor -- start 2023-07-20T19:01:43.973 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 -- start start 2023-07-20T19:01:43.973 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fca200ff290 0x7fca200692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 0x7fca2019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca2019acd0 0x7fca2019d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2019d900 con 0x7fca2019acd0 2023-07-20T19:01:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2019da70 con 0x7fca200ff290 2023-07-20T19:01:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca2788d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2019dc30 con 0x7fca20198350 2023-07-20T19:01:43.974 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca24e28700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 0x7fca2019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca24e28700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 0x7fca2019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48348/0 (socket says 172.21.15.156:48348) 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca24e28700 1 -- 172.21.15.156:0/1129892986 learned_addr learned my addr 172.21.15.156:0/1129892986 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.917+0000 7fca25e2a700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca2019acd0 0x7fca2019d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca25629700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fca200ff290 0x7fca200692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca24e28700 1 -- 172.21.15.156:0/1129892986 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fca200ff290 msgr2=0x7fca200692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca24e28700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fca200ff290 0x7fca200692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:43.975 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca24e28700 1 -- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca2019acd0 msgr2=0x7fca2019d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca24e28700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca2019acd0 0x7fca2019d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca24e28700 1 -- 172.21.15.156:0/1129892986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c0095f0 con 0x7fca20198350 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca25e2a700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca2019acd0 0x7fca2019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca24e28700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 0x7fca2019a740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b780 tx=0x7fca1000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10004230 con 0x7fca20198350 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca10004390 con 0x7fca20198350 2023-07-20T19:01:43.976 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca20194250 con 0x7fca20198350 2023-07-20T19:01:43.977 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10011ad0 con 0x7fca20198350 2023-07-20T19:01:43.977 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.918+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca201947d0 con 0x7fca20198350 2023-07-20T19:01:43.977 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.919+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca20104f90 con 0x7fca20198350 2023-07-20T19:01:43.977 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.920+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fca100036a0 con 0x7fca20198350 2023-07-20T19:01:43.978 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.920+0000 7fca167fc700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fca0c038c30 0x7fca0c03b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:43.978 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.920+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fca1004b680 con 0x7fca20198350 2023-07-20T19:01:43.978 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.920+0000 7fca25629700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fca0c038c30 0x7fca0c03b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:43.978 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.920+0000 7fca25629700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fca0c038c30 0x7fca0c03b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca1c005130 tx=0x7fca1c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:43.979 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:43.924+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fca10004500 con 0x7fca20198350 2023-07-20T19:01:44.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.129+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi061=x;smithi149=y;smithi156=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fca2018d8d0 con 0x7fca0c038c30 2023-07-20T19:01:44.191 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.137+0000 7fca167fc700 1 -- 172.21.15.156:0/1129892986 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fca2018d8d0 con 0x7fca0c038c30 2023-07-20T19:01:44.191 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mgr update... 2023-07-20T19:01:44.193 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fca0c038c30 msgr2=0x7fca0c03b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:44.193 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fca0c038c30 0x7fca0c03b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca1c005130 tx=0x7fca1c00b540 comp rx=0 tx=0).stop 2023-07-20T19:01:44.193 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 msgr2=0x7fca2019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:44.193 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 0x7fca2019a740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b780 tx=0x7fca1000ba90 comp rx=0 tx=0).stop 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 shutdown_connections 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fca0c038c30 0x7fca0c03b0f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca2019acd0 0x7fca2019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fca200ff290 0x7fca200692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 --2- 172.21.15.156:0/1129892986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca20198350 0x7fca2019a740 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 >> 172.21.15.156:0/1129892986 conn(0x7fca200fa810 msgr2=0x7fca200fb550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:44.194 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 shutdown_connections 2023-07-20T19:01:44.195 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:01:44.139+0000 7fca2788d700 1 -- 172.21.15.156:0/1129892986 wait complete. 2023-07-20T19:01:44.515 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:44.515 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:44.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:44.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:44.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.681 DEBUG:teuthology.orchestra.run.smithi149:mgr.y> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T19:01:44.684 DEBUG:teuthology.orchestra.run.smithi156:mgr.z> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T19:01:44.686 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T19:01:44.686 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-07-20T19:01:44.687 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T19:01:44.708 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T19:01:44.708 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_1 2023-07-20T19:01:44.710 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:44.711 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.711 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.711 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:44.711 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:44.711 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:44.711 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:44.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T19:01:44.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T19:01:44.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:44.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:44.716 INFO:journalctl@ceph.mgr.y.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:01:44.717 INFO:journalctl@ceph.mgr.z.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:01:44.735 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T19:01:44.736 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:44.736 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 381520 Links: 1 2023-07-20T19:01:44.736 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:44.736 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:44.736 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-07-20 19:00:57.903337285 +0000 2023-07-20T19:01:44.737 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-07-20 18:47:32.367309251 +0000 2023-07-20T19:01:44.737 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-07-20 18:47:32.367309251 +0000 2023-07-20T19:01:44.737 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2023-07-20T19:01:44.737 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T19:01:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-07-20T19:01:44.808 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-07-20T19:01:44.808 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000252561 s, 2.0 MB/s 2023-07-20T19:01:44.809 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T19:01:44.870 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_2 2023-07-20T19:01:44.930 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T19:01:44.930 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:44.931 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 380211 Links: 1 2023-07-20T19:01:44.931 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:44.931 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:44.931 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-07-20 19:00:57.903337285 +0000 2023-07-20T19:01:44.931 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-07-20 18:47:32.916300425 +0000 2023-07-20T19:01:44.932 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-07-20 18:47:32.916300425 +0000 2023-07-20T19:01:44.932 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2023-07-20T19:01:44.932 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T19:01:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-07-20T19:01:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-07-20T19:01:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000214034 s, 2.4 MB/s 2023-07-20T19:01:44.999 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T19:01:45.058 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_3 2023-07-20T19:01:45.116 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T19:01:45.117 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:45.117 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 377330 Links: 1 2023-07-20T19:01:45.117 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:45.117 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:45.117 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-07-20 19:00:57.903337285 +0000 2023-07-20T19:01:45.118 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-07-20 18:47:33.431292146 +0000 2023-07-20T19:01:45.118 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-07-20 18:47:33.431292146 +0000 2023-07-20T19:01:45.118 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2023-07-20T19:01:45.118 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T19:01:45.182 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-07-20T19:01:45.183 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-07-20T19:01:45.183 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000184329 s, 2.8 MB/s 2023-07-20T19:01:45.184 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T19:01:45.243 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_4 2023-07-20T19:01:45.301 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T19:01:45.301 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:45.301 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 380428 Links: 1 2023-07-20T19:01:45.301 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:45.301 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:45.301 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-07-20 19:00:57.903337285 +0000 2023-07-20T19:01:45.302 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-07-20 18:47:33.964283578 +0000 2023-07-20T19:01:45.302 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-07-20 18:47:33.964283578 +0000 2023-07-20T19:01:45.302 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2023-07-20T19:01:45.302 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T19:01:45.367 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-07-20T19:01:45.367 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-07-20T19:01:45.367 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000224651 s, 2.3 MB/s 2023-07-20T19:01:45.368 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T19:01:45.426 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-07-20T19:01:45.426 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T19:01:45.447 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T19:01:45.447 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2023-07-20T19:01:45.507 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T19:01:45.507 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:45.507 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 381969 Links: 1 2023-07-20T19:01:45.508 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:45.508 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:45.508 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-07-20 19:01:32.953961395 +0000 2023-07-20T19:01:45.508 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-07-20 18:54:00.819029856 +0000 2023-07-20T19:01:45.508 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-07-20 18:54:00.819029856 +0000 2023-07-20T19:01:45.508 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-07-20T19:01:45.509 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T19:01:45.576 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-07-20T19:01:45.576 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-07-20T19:01:45.576 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000340551 s, 1.5 MB/s 2023-07-20T19:01:45.577 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T19:01:45.636 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2023-07-20T19:01:45.694 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T19:01:45.694 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:45.694 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 380587 Links: 1 2023-07-20T19:01:45.695 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:45.695 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:45.695 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-07-20 19:01:32.953961395 +0000 2023-07-20T19:01:45.695 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-07-20 18:54:01.358018295 +0000 2023-07-20T19:01:45.695 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-07-20 18:54:01.358018295 +0000 2023-07-20T19:01:45.695 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-07-20T19:01:45.696 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T19:01:45.698 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: Reconfiguring mon.c (monmap changed)... 2023-07-20T19:01:45.698 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: Reconfiguring daemon mon.c on smithi156 2023-07-20T19:01:45.698 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi061=x;smithi149=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:45.698 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: Saving service mgr spec with placement smithi061=x;smithi149=y;smithi156=z;count:3 2023-07-20T19:01:45.698 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:45.699 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T19:01:45.700 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:45.700 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:45.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: Reconfiguring mon.c (monmap changed)... 2023-07-20T19:01:45.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: Reconfiguring daemon mon.c on smithi156 2023-07-20T19:01:45.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi061=x;smithi149=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:45.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: Saving service mgr spec with placement smithi061=x;smithi149=y;smithi156=z;count:3 2023-07-20T19:01:45.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T19:01:45.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:45.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:45.758 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: Reconfiguring mon.c (monmap changed)... 2023-07-20T19:01:45.758 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: Reconfiguring daemon mon.c on smithi156 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi061=x;smithi149=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: Saving service mgr spec with placement smithi061=x;smithi149=y;smithi156=z;count:3 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:45.759 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:45.760 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:45.760 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:45.760 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T19:01:45.760 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:45.760 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:45.761 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-07-20T19:01:45.761 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-07-20T19:01:45.761 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000222482 s, 2.3 MB/s 2023-07-20T19:01:45.762 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T19:01:45.820 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2023-07-20T19:01:45.877 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 381415 Links: 1 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-07-20 19:01:32.953961395 +0000 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-07-20 18:54:01.934005941 +0000 2023-07-20T19:01:45.878 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-07-20 18:54:01.934005941 +0000 2023-07-20T19:01:45.879 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-07-20T19:01:45.879 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T19:01:45.943 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-07-20T19:01:45.943 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-07-20T19:01:45.943 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000240438 s, 2.1 MB/s 2023-07-20T19:01:45.944 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T19:01:46.002 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2023-07-20T19:01:46.059 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T19:01:46.059 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:46.059 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 381497 Links: 1 2023-07-20T19:01:46.060 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:46.060 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:46.060 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-07-20 19:01:32.953961395 +0000 2023-07-20T19:01:46.060 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-07-20 18:54:02.448994895 +0000 2023-07-20T19:01:46.060 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-07-20 18:54:02.448994895 +0000 2023-07-20T19:01:46.060 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-07-20T19:01:46.061 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T19:01:46.124 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-07-20T19:01:46.124 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-07-20T19:01:46.124 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000183225 s, 2.8 MB/s 2023-07-20T19:01:46.125 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T19:01:46.183 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-20T19:01:46.183 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T19:01:46.199 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T19:01:46.200 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2023-07-20T19:01:46.257 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T19:01:46.257 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:46.257 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 377619 Links: 1 2023-07-20T19:01:46.257 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:46.257 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:46.257 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-20 19:01:24.665842090 +0000 2023-07-20T19:01:46.258 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-20 18:49:40.150015944 +0000 2023-07-20T19:01:46.258 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-20 18:49:40.150015944 +0000 2023-07-20T19:01:46.258 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-20T19:01:46.258 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T19:01:46.371 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-20T19:01:46.372 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-20T19:01:46.372 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000117753 s, 4.3 MB/s 2023-07-20T19:01:46.373 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T19:01:46.390 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2023-07-20T19:01:46.448 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T19:01:46.449 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:46.449 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 378266 Links: 1 2023-07-20T19:01:46.449 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:46.449 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:46.450 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-20 19:01:46.265522636 +0000 2023-07-20T19:01:46.450 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-20 18:49:40.653008621 +0000 2023-07-20T19:01:46.450 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-20 18:49:40.653008621 +0000 2023-07-20T19:01:46.450 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-20T19:01:46.450 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T19:01:46.518 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-20T19:01:46.518 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-20T19:01:46.518 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.00012582 s, 4.1 MB/s 2023-07-20T19:01:46.519 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T19:01:46.580 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2023-07-20T19:01:46.638 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T19:01:46.638 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 380245 Links: 1 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-20 19:01:46.395520706 +0000 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-20 18:49:41.186000861 +0000 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-20 18:49:41.186000861 +0000 2023-07-20T19:01:46.639 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-20T19:01:46.640 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T19:01:46.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:46 smithi156 ceph-mon[129228]: Deploying daemon mgr.z on smithi156 2023-07-20T19:01:46.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:46 smithi156 ceph-mon[129228]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:46.663 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:46 smithi156 systemd[1]: Starting Ceph mgr.z for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-20T19:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-20T19:01:46.693 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000122498 s, 4.2 MB/s 2023-07-20T19:01:46.693 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T19:01:46.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:46 smithi061 ceph-mon[127454]: Deploying daemon mgr.z on smithi156 2023-07-20T19:01:46.715 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:46 smithi061 ceph-mon[127454]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:46.751 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2023-07-20T19:01:46.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:46 smithi149 ceph-mon[128885]: Deploying daemon mgr.z on smithi156 2023-07-20T19:01:46.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:46 smithi149 ceph-mon[128885]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:46.808 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T19:01:46.809 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T19:01:46.809 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 378449 Links: 1 2023-07-20T19:01:46.809 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T19:01:46.809 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T19:01:46.810 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-20 19:01:46.395520706 +0000 2023-07-20T19:01:46.810 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-20 18:49:41.719993086 +0000 2023-07-20T19:01:46.810 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-20 18:49:41.719993086 +0000 2023-07-20T19:01:46.810 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-20T19:01:46.810 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T19:01:46.878 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-20T19:01:46.878 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-20T19:01:46.878 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000135023 s, 3.8 MB/s 2023-07-20T19:01:46.879 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T19:01:46.936 INFO:tasks.cephadm:Deploying osd.0 on smithi061 with /dev/vg_nvme/lv_4... 2023-07-20T19:01:46.937 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T19:01:47.027 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:46 smithi156 podman[131254]: 2023-07-20T19:01:47.327 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:46 smithi156 bash[131254]: f6d9136e6df262496be2378efa7ea30c6988a319e7970e24b02f44707b4265b3 2023-07-20T19:01:47.327 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:46 smithi156 ceph-mgr[131399]: -- 172.21.15.156:0/3455450175 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5612bd3511e0 con 0x5612bd327000 2023-07-20T19:01:47.327 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 systemd[1]: Started Ceph mgr.z for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:01:47.328 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.105+0000 7ff209eb6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T19:01:47.328 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.185+0000 7ff209eb6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T19:01:47.328 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.273+0000 7ff209eb6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T19:01:47.507 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:47.588 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.365+0000 7ff209eb6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T19:01:47.892 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.534+0000 7ff209eb6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T19:01:48.148 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.838+0000 7ff209eb6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T19:01:48.148 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:47.926+0000 7ff209eb6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T19:01:48.148 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:48.056+0000 7ff209eb6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T19:01:48.412 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:48.151+0000 7ff209eb6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T19:01:48.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:48.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T19:01:48.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:48.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:48.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: Deploying daemon mgr.y on smithi149 2023-07-20T19:01:48.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-mon[129228]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:48.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:48.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T19:01:48.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:48.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:48.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: Deploying daemon mgr.y on smithi149 2023-07-20T19:01:48.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:48 smithi149 ceph-mon[128885]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:48.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:48.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:48.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T19:01:48.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:48.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:48.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: Deploying daemon mgr.y on smithi149 2023-07-20T19:01:48.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:48 smithi061 ceph-mon[127454]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:48.912 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:48.569+0000 7ff209eb6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T19:01:49.020 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T19:01:49.021 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T19:01:49.021 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 10+0 records in 2023-07-20T19:01:49.021 INFO:teuthology.orchestra.run.smithi061.stderr:10+0 records out 2023-07-20T19:01:49.021 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127817 s, 820 MB/s 2023-07-20T19:01:49.021 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping successful for: 2023-07-20T19:01:49.052 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:48 smithi149 systemd[1]: Starting Ceph mgr.y for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:01:49.342 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 podman[130318]: 2023-07-20T19:01:49.456 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi061:vg_nvme/lv_4 2023-07-20T19:01:49.642 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 bash[130318]: add4180d2ec4496236336e766029087b3737014b3cd4c868ac5be5553a26d5e9 2023-07-20T19:01:49.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 ceph-mgr[130359]: -- 172.21.15.149:0/1641259381 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5616b2c6d1e0 con 0x5616b2c43000 2023-07-20T19:01:49.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 systemd[1]: Started Ceph mgr.y for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:01:49.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:49.416+0000 7f6224828200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T19:01:49.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:49.495+0000 7f6224828200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T19:01:49.643 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:49.588+0000 7f6224828200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T19:01:49.662 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:49.139+0000 7ff209eb6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T19:01:49.662 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:49.309+0000 7ff209eb6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T19:01:49.920 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:49.685+0000 7f6224828200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T19:01:49.921 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:49.865+0000 7f6224828200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T19:01:50.023 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:01:50.412 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:50.102+0000 7ff209eb6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:01:50.413 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:50.199+0000 7ff209eb6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T19:01:50.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:50 smithi061 ceph-mon[127454]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:50.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:50.551 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:50.194+0000 7f6224828200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T19:01:50.551 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:50.288+0000 7f6224828200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T19:01:50.552 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:50.424+0000 7f6224828200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T19:01:50.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-mon[128885]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:50.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:50.723 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:50.376+0000 7ff209eb6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:01:50.723 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.723 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.723 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.724 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:50.724 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-mon[129228]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:50.724 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:51.023 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:50.519+0000 7f6224828200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T19:01:51.023 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:50.967+0000 7f6224828200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T19:01:51.120 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:50.668+0000 7ff209eb6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T19:01:51.121 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:50.912+0000 7ff209eb6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T19:01:51.358 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.303+0000 7f5d537b3700 1 -- 172.21.15.61:0/4023428918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c101a00 msgr2=0x7f5d4c101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:51.358 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.303+0000 7f5d537b3700 1 --2- 172.21.15.61:0/4023428918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c101a00 0x7f5d4c101e60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0099c0 tx=0x7f5d3c009cd0 comp rx=0 tx=0).stop 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 -- 172.21.15.61:0/4023428918 shutdown_connections 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 --2- 172.21.15.61:0/4023428918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 --2- 172.21.15.61:0/4023428918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5d4c100800 0x7f5d4c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 --2- 172.21.15.61:0/4023428918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c101a00 0x7f5d4c101e60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 -- 172.21.15.61:0/4023428918 >> 172.21.15.61:0/4023428918 conn(0x7f5d4c0fbd60 msgr2=0x7f5d4c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 -- 172.21.15.61:0/4023428918 shutdown_connections 2023-07-20T19:01:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 -- 172.21.15.61:0/4023428918 wait complete. 2023-07-20T19:01:51.360 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.304+0000 7f5d537b3700 1 Processor -- start 2023-07-20T19:01:51.361 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 -- start start 2023-07-20T19:01:51.361 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d4c100800 0x7f5d4c1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:51.361 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5d4c101a00 0x7f5d4c196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:51.361 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c19daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:51.361 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c19e070 con 0x7f5d4c1023a0 2023-07-20T19:01:51.361 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c19e1b0 con 0x7f5d4c101a00 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.305+0000 7f5d537b3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c19e340 con 0x7f5d4c100800 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c19daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d5154f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d4c100800 0x7f5d4c1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c19daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57580/0 (socket says 172.21.15.61:57580) 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 -- 172.21.15.61:0/370314927 learned_addr learned my addr 172.21.15.61:0/370314927 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d50d4e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5d4c101a00 0x7f5d4c196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:51.362 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 -- 172.21.15.61:0/370314927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d4c100800 msgr2=0x7f5d4c1964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d4c100800 0x7f5d4c1964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 -- 172.21.15.61:0/370314927 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5d4c101a00 msgr2=0x7f5d4c196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5d4c101a00 0x7f5d4c196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 -- 172.21.15.61:0/370314927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0095f0 con 0x7f5d4c1023a0 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d51d50700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c19daa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5d44007f10 tx=0x7f5d4400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4400dd20 con 0x7f5d4c1023a0 2023-07-20T19:01:51.363 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c19e620 con 0x7f5d4c1023a0 2023-07-20T19:01:51.364 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d4400f040 con 0x7f5d4c1023a0 2023-07-20T19:01:51.364 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d440048a0 con 0x7f5d4c1023a0 2023-07-20T19:01:51.364 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.306+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c19ead0 con 0x7f5d4c1023a0 2023-07-20T19:01:51.365 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.307+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f5d44015400 con 0x7f5d4c1023a0 2023-07-20T19:01:51.365 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.307+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c10c000 con 0x7f5d4c1023a0 2023-07-20T19:01:51.365 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.307+0000 7f5d427fc700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5d380389e0 0x7f5d3803aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:01:51.365 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.307+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5d4404aae0 con 0x7f5d4c1023a0 2023-07-20T19:01:51.365 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.308+0000 7f5d5154f700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5d380389e0 0x7f5d3803aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:01:51.365 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.308+0000 7f5d5154f700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5d380389e0 0x7f5d3803aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c101860 tx=0x7f5d4800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:01:51.367 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.312+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d440138f0 con 0x7f5d4c1023a0 2023-07-20T19:01:51.388 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:51.066+0000 7ff209eb6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T19:01:51.578 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:51.523+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d4c060bb0 con 0x7f5d380389e0 2023-07-20T19:01:51.662 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:51.334+0000 7ff209eb6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T19:01:52.049 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:51.557+0000 7f6224828200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T19:01:52.049 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:51.730+0000 7f6224828200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T19:01:52.407 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:52.034+0000 7ff209eb6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T19:01:52.408 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:52.164+0000 7ff209eb6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T19:01:52.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.408 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.409 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: Reconfiguring daemon mgr.x on smithi061 2023-07-20T19:01:52.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:52.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:01:52.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:01:52.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.410 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-mon[129228]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:52.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.465 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:52.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: Reconfiguring daemon mgr.x on smithi061 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:52 smithi061 ceph-mon[127454]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:52.549 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.549 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T19:01:52.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: Reconfiguring daemon mgr.x on smithi061 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:01:52.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:52.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-mon[128885]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:52.662 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:52.353+0000 7ff209eb6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T19:01:52.663 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:52.467+0000 7ff209eb6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T19:01:52.813 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:52.497+0000 7f6224828200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:01:52.813 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:52.587+0000 7f6224828200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T19:01:53.162 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:52.681+0000 7ff209eb6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T19:01:53.468 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:52.758+0000 7f6224828200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T19:01:53.468 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:53.034+0000 7f6224828200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T19:01:53.468 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:53.266+0000 7f6224828200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T19:01:53.469 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:53.413+0000 7f6224828200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T19:01:53.912 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z[131332]: 2023-07-20T19:01:53.536+0000 7ff209eb6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T19:01:53.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:53.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:53.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:53.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: Standby manager daemon z started 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T19:01:53.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T19:01:53.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:53 smithi156 ceph-mon[129228]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:54.049 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:53.677+0000 7f6224828200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:54.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: Standby manager daemon z started 2023-07-20T19:01:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T19:01:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T19:01:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T19:01:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T19:01:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:53 smithi149 ceph-mon[128885]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:54.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:01:54.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: Standby manager daemon z started 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.156:0/1021513137' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T19:01:54.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:53 smithi061 ceph-mon[127454]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:54.514 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:54.458+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f5d440143a0 con 0x7f5d4c1023a0 2023-07-20T19:01:54.726 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:54.387+0000 7f6224828200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T19:01:54.727 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:54.520+0000 7f6224828200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T19:01:55.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/440039522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfae06af-18f4-487e-ae4b-8e8c9d36ef76"}]: dispatch 2023-07-20T19:01:55.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/440039522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfae06af-18f4-487e-ae4b-8e8c9d36ef76"}]': finished 2023-07-20T19:01:55.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-mon[128885]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T19:01:55.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:01:55.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-mon[128885]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-20T19:01:55.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T19:01:55.050 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:54.713+0000 7f6224828200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T19:01:55.051 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:54.826+0000 7f6224828200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T19:01:55.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:54 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/440039522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfae06af-18f4-487e-ae4b-8e8c9d36ef76"}]: dispatch 2023-07-20T19:01:55.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:54 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/440039522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfae06af-18f4-487e-ae4b-8e8c9d36ef76"}]': finished 2023-07-20T19:01:55.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:54 smithi156 ceph-mon[129228]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T19:01:55.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:01:55.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:54 smithi156 ceph-mon[129228]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-20T19:01:55.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T19:01:55.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:54 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/440039522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfae06af-18f4-487e-ae4b-8e8c9d36ef76"}]: dispatch 2023-07-20T19:01:55.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:54 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/440039522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfae06af-18f4-487e-ae4b-8e8c9d36ef76"}]': finished 2023-07-20T19:01:55.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:54 smithi061 ceph-mon[127454]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T19:01:55.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:01:55.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:54 smithi061 ceph-mon[127454]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-20T19:01:55.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T19:01:55.549 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:55.041+0000 7f6224828200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T19:01:56.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:55 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1847643072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:01:56.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:55 smithi149 ceph-mon[128885]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:56.050 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:01:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y[130334]: 2023-07-20T19:01:55.917+0000 7f6224828200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T19:01:56.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:55 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1847643072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:01:56.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:55 smithi156 ceph-mon[129228]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:56.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:55 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1847643072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:01:56.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:55 smithi061 ceph-mon[127454]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:56.743 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:01:56.689+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f5d4401c310 con 0x7f5d4c1023a0 2023-07-20T19:01:57.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:56 smithi149 ceph-mon[128885]: Standby manager daemon y started 2023-07-20T19:01:57.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:56 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T19:01:57.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:56 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T19:01:57.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:56 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T19:01:57.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:56 smithi149 ceph-mon[128885]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T19:01:57.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:56 smithi156 ceph-mon[129228]: Standby manager daemon y started 2023-07-20T19:01:57.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:56 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T19:01:57.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:56 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T19:01:57.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:56 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T19:01:57.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:56 smithi156 ceph-mon[129228]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T19:01:57.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:56 smithi061 ceph-mon[127454]: Standby manager daemon y started 2023-07-20T19:01:57.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:56 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T19:01:57.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:56 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T19:01:57.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:56 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T19:01:57.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:56 smithi061 ceph-mon[127454]: from='mgr.? 172.21.15.149:0/486710910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T19:01:58.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:57 smithi149 ceph-mon[128885]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-20T19:01:58.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T19:01:58.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:57 smithi149 ceph-mon[128885]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:58.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:57 smithi156 ceph-mon[129228]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-20T19:01:58.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T19:01:58.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:57 smithi156 ceph-mon[129228]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:01:58.198 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:57 smithi061 ceph-mon[127454]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-20T19:01:58.198 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T19:01:58.198 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:57 smithi061 ceph-mon[127454]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:00.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:01:59 smithi149 ceph-mon[128885]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:00.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:01:59 smithi156 ceph-mon[129228]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:00.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:01:59 smithi061 ceph-mon[127454]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:00.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T19:02:00.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:00.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:00 smithi061 ceph-mon[127454]: Deploying daemon osd.0 on smithi061 2023-07-20T19:02:01.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T19:02:01.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:01.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:00 smithi149 ceph-mon[128885]: Deploying daemon osd.0 on smithi061 2023-07-20T19:02:01.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T19:02:01.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:01.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:00 smithi156 ceph-mon[129228]: Deploying daemon osd.0 on smithi061 2023-07-20T19:02:01.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:01 smithi061 ceph-mon[127454]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:02.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:01 smithi149 ceph-mon[128885]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:02.162 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:01 smithi156 ceph-mon[129228]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:03.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:03 smithi061 ceph-mon[127454]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:04.049 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:03 smithi149 ceph-mon[128885]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:04.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:03 smithi156 ceph-mon[129228]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:06.177 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:05 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:06.178 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:05 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:06.178 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:05 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:06.178 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:05 smithi061 ceph-mon[127454]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:06.299 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:05 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:06.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:05 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:06.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:05 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:06.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:05 smithi149 ceph-mon[128885]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:06.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:05 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:06.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:05 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:06.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:05 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:06.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:05 smithi156 ceph-mon[129228]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.730+0000 7f5d427fc700 1 -- 172.21.15.61:0/370314927 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5d4c060bb0 con 0x7f5d380389e0 2023-07-20T19:02:07.785 INFO:teuthology.orchestra.run.smithi061.stdout:Created osd(s) 0 on host 'smithi061' 2023-07-20T19:02:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5d380389e0 msgr2=0x7f5d3803aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5d380389e0 0x7f5d3803aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c101860 tx=0x7f5d4800b410 comp rx=0 tx=0).stop 2023-07-20T19:02:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 msgr2=0x7f5d4c19daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c19daa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5d44007f10 tx=0x7f5d4400d3b0 comp rx=0 tx=0).stop 2023-07-20T19:02:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 shutdown_connections 2023-07-20T19:02:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5d380389e0 0x7f5d3803aea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d4c1023a0 0x7f5d4c19daa0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d4c100800 0x7f5d4c1964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 --2- 172.21.15.61:0/370314927 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5d4c101a00 0x7f5d4c196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.732+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 >> 172.21.15.61:0/370314927 conn(0x7f5d4c0fbd60 msgr2=0x7f5d4c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:02:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.733+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 shutdown_connections 2023-07-20T19:02:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:07.733+0000 7f5d537b3700 1 -- 172.21.15.61:0/370314927 wait complete. 2023-07-20T19:02:07.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:07.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:07.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:07.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:07.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:07.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:07.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:07.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:07.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:08.127 DEBUG:teuthology.orchestra.run.smithi061:osd.0> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T19:02:08.129 INFO:tasks.cephadm:Deploying osd.1 on smithi149 with /dev/vg_nvme/lv_4... 2023-07-20T19:02:08.130 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T19:02:08.162 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.162 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.162 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:08.162 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:08.162 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:08.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:08.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:08.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:08.165 INFO:journalctl@ceph.osd.0.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:02:08.693 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:02:09.189 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0[140435]: 2023-07-20T19:02:08.840+0000 7fba3f2dc640 -1 osd.0 0 log_to_monitors true 2023-07-20T19:02:09.766 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:09 smithi149 ceph-mon[128885]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T19:02:09.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:09 smithi156 ceph-mon[129228]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T19:02:09.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:09 smithi061 ceph-mon[127454]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T19:02:10.200 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T19:02:10.201 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T19:02:10.201 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2023-07-20T19:02:10.201 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2023-07-20T19:02:10.201 INFO:teuthology.orchestra.run.smithi149.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145586 s, 720 MB/s 2023-07-20T19:02:10.201 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 2023-07-20T19:02:10.201 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2023-07-20T19:02:10.635 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi149:vg_nvme/lv_4 2023-07-20T19:02:10.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T19:02:10.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T19:02:10.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:10.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:02:10.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:10.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:10.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:10.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:10.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.718 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0[140435]: 2023-07-20T19:02:10.474+0000 7fba34954700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T19:02:10.718 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0[140435]: 2023-07-20T19:02:10.477+0000 7fba2ef47700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:02:10.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T19:02:10.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T19:02:10.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:10.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:02:10.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:10.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:10.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:10.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:10.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T19:02:10.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T19:02:10.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:10.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:02:10.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:10.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:11.226 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:02:11.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:11 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi061 2023-07-20T19:02:11.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:11 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi061 to 17168M 2023-07-20T19:02:11.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:11 smithi149 ceph-mon[128885]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:02:11.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:11 smithi149 ceph-mon[128885]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T19:02:11.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:11.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:11.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:11 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi061 2023-07-20T19:02:11.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:11 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi061 to 17168M 2023-07-20T19:02:11.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:11 smithi156 ceph-mon[129228]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:02:11.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:11 smithi156 ceph-mon[129228]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T19:02:11.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:11.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:11.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:11 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi061 2023-07-20T19:02:11.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:11 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi061 to 17168M 2023-07-20T19:02:11.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:11 smithi061 ceph-mon[127454]: from='osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:02:11.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:11 smithi061 ceph-mon[127454]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T19:02:11.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:11.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:12.568 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.513+0000 7fdc72d82700 1 -- 172.21.15.149:0/1271556640 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 msgr2=0x7fdc6c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:12.568 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.513+0000 7fdc72d82700 1 --2- 172.21.15.149:0/1271556640 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c0ffb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc600099c0 tx=0x7fdc60009cd0 comp rx=0 tx=0).stop 2023-07-20T19:02:12.568 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- 172.21.15.149:0/1271556640 shutdown_connections 2023-07-20T19:02:12.568 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 --2- 172.21.15.149:0/1271556640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc6c1012a0 0x7fdc6c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:12.568 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 --2- 172.21.15.149:0/1271556640 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c0ffb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:12.569 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 --2- 172.21.15.149:0/1271556640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc6c100900 0x7fdc6c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:12.569 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- 172.21.15.149:0/1271556640 >> 172.21.15.149:0/1271556640 conn(0x7fdc6c0fac80 msgr2=0x7fdc6c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:02:12.569 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- 172.21.15.149:0/1271556640 shutdown_connections 2023-07-20T19:02:12.569 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- 172.21.15.149:0/1271556640 wait complete. 2023-07-20T19:02:12.571 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 Processor -- start 2023-07-20T19:02:12.571 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- start start 2023-07-20T19:02:12.571 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:12.571 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc6c100900 0x7fdc6c195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc6c1012a0 0x7fdc6c19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19cfd0 con 0x7fdc6c100900 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19d110 con 0x7fdc6c0ff700 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc72d82700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19d2a0 con 0x7fdc6c1012a0 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc70b1e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.514+0000 7fdc70b1e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:44050/0 (socket says 172.21.15.149:44050) 2023-07-20T19:02:12.572 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 -- 172.21.15.149:0/836698089 learned_addr learned my addr 172.21.15.149:0/836698089 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc7131f700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc6c1012a0 0x7fdc6c19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc6bfff700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc6c100900 0x7fdc6c195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 -- 172.21.15.149:0/836698089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc6c1012a0 msgr2=0x7fdc6c19ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc6c1012a0 0x7fdc6c19ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 -- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc6c100900 msgr2=0x7fdc6c195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc6c100900 0x7fdc6c195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 -- 172.21.15.149:0/836698089 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc600095f0 con 0x7fdc6c0ff700 2023-07-20T19:02:12.573 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc70b1e700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c195440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdc600036f0 tx=0x7fdc60009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:02:12.574 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mon.2 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6001d070 con 0x7fdc6c0ff700 2023-07-20T19:02:12.574 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c068370 con 0x7fdc6c0ff700 2023-07-20T19:02:12.574 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mon.2 v2:172.21.15.149:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc60004010 con 0x7fdc6c0ff700 2023-07-20T19:02:12.574 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mon.2 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000fe30 con 0x7fdc6c0ff700 2023-07-20T19:02:12.574 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.515+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c068810 con 0x7fdc6c0ff700 2023-07-20T19:02:12.574 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.516+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c10af60 con 0x7fdc6c0ff700 2023-07-20T19:02:12.575 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.517+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mon.2 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fdc6000f560 con 0x7fdc6c0ff700 2023-07-20T19:02:12.575 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.518+0000 7fdc69ffb700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fdc5c0a10c0 0x7fdc5c0a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:12.575 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.518+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mon.2 v2:172.21.15.149:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fdc600cbe10 con 0x7fdc6c0ff700 2023-07-20T19:02:12.575 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.518+0000 7fdc6bfff700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fdc5c0a10c0 0x7fdc5c0a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:12.576 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.519+0000 7fdc6bfff700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fdc5c0a10c0 0x7fdc5c0a3580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800bbd0 tx=0x7fdc5800b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:02:12.576 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.522+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mon.2 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdc60096fc0 con 0x7fdc6c0ff700 2023-07-20T19:02:12.767 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:12.713+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc6c060bb0 con 0x7fdc5c0a10c0 2023-07-20T19:02:12.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:12 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:02:12.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:12 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:02:12.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:12 smithi149 ceph-mon[128885]: osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099] boot 2023-07-20T19:02:12.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:12 smithi149 ceph-mon[128885]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T19:02:12.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:12 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:12.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:12 smithi149 ceph-mon[128885]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:12.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:12 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:02:12.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:12 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:02:12.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:12 smithi156 ceph-mon[129228]: osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099] boot 2023-07-20T19:02:12.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:12 smithi156 ceph-mon[129228]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T19:02:12.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:12 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:12.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:12 smithi156 ceph-mon[129228]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:12.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:12 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:02:12.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:12 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:02:12.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:12 smithi061 ceph-mon[127454]: osd.0 [v2:172.21.15.61:6802/1072462099,v1:172.21.15.61:6803/1072462099] boot 2023-07-20T19:02:12.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:12 smithi061 ceph-mon[127454]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T19:02:12.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:12 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T19:02:12.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:12 smithi061 ceph-mon[127454]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T19:02:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:13 smithi149 ceph-mon[128885]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T19:02:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:13.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:13 smithi156 ceph-mon[129228]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T19:02:13.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:13.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:13.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:13.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:13 smithi061 ceph-mon[127454]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T19:02:13.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:13.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:13.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:14.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:14 smithi149 ceph-mon[128885]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:14.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:14 smithi149 ceph-mon[128885]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:14.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:14 smithi156 ceph-mon[129228]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:14.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:14 smithi156 ceph-mon[129228]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:14.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:14 smithi061 ceph-mon[127454]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:14.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:14 smithi061 ceph-mon[127454]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:15.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:15 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/1816575706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]: dispatch 2023-07-20T19:02:15.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:15 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]: dispatch 2023-07-20T19:02:15.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:15 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]': finished 2023-07-20T19:02:15.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:15 smithi149 ceph-mon[128885]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T19:02:15.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:15.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:15 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/1816575706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]: dispatch 2023-07-20T19:02:15.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:15 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]: dispatch 2023-07-20T19:02:15.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:15 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]': finished 2023-07-20T19:02:15.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:15 smithi156 ceph-mon[129228]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T19:02:15.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:15.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:15 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/1816575706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]: dispatch 2023-07-20T19:02:15.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:15 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]: dispatch 2023-07-20T19:02:15.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:15 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3e2b1d4-abe5-4438-9196-6ef26230a071"}]': finished 2023-07-20T19:02:15.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:15 smithi061 ceph-mon[127454]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T19:02:15.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:16.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:16 smithi156 ceph-mon[129228]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:16.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:16 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/3765016960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:16.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:16 smithi061 ceph-mon[127454]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:16.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:16 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/3765016960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:17.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:16 smithi149 ceph-mon[128885]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:17.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:16 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/3765016960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:18.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:18 smithi149 ceph-mon[128885]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:18.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:18 smithi156 ceph-mon[129228]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:18.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:18 smithi061 ceph-mon[127454]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:20.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:19 smithi149 ceph-mon[128885]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:20.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:19 smithi156 ceph-mon[129228]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:20.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:19 smithi061 ceph-mon[127454]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:22.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:21 smithi149 ceph-mon[128885]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:22.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:21 smithi156 ceph-mon[129228]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:22.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:21 smithi061 ceph-mon[127454]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:22.798 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T19:02:22.798 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:22.798 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:22 smithi149 ceph-mon[128885]: Deploying daemon osd.1 on smithi149 2023-07-20T19:02:23.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T19:02:23.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:23.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:22 smithi156 ceph-mon[129228]: Deploying daemon osd.1 on smithi149 2023-07-20T19:02:23.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T19:02:23.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:23.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:22 smithi061 ceph-mon[127454]: Deploying daemon osd.1 on smithi149 2023-07-20T19:02:24.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:23 smithi149 ceph-mon[128885]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:24.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:23 smithi156 ceph-mon[129228]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:24.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:23 smithi061 ceph-mon[127454]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:26.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:25 smithi149 ceph-mon[128885]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:26.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:25 smithi156 ceph-mon[129228]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:26.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:25 smithi061 ceph-mon[127454]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:27.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:27 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:27.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:27 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:27.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:27 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:27.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:27 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:27.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:27 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:27.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:27 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:27.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:27 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:27.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:27 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:27.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:27 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:28.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:28 smithi149 ceph-mon[128885]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:28.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:28 smithi156 ceph-mon[129228]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:28.965 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:28 smithi061 ceph-mon[127454]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:29.487 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 1 on host 'smithi149' 2023-07-20T19:02:29.487 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.430+0000 7fdc69ffb700 1 -- 172.21.15.149:0/836698089 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdc6c060bb0 con 0x7fdc5c0a10c0 2023-07-20T19:02:29.488 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fdc5c0a10c0 msgr2=0x7fdc5c0a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:29.489 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fdc5c0a10c0 0x7fdc5c0a3580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800bbd0 tx=0x7fdc5800b5b0 comp rx=0 tx=0).stop 2023-07-20T19:02:29.489 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 msgr2=0x7fdc6c195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:29.489 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c195440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdc600036f0 tx=0x7fdc60009580 comp rx=0 tx=0).stop 2023-07-20T19:02:29.489 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 shutdown_connections 2023-07-20T19:02:29.489 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fdc5c0a10c0 0x7fdc5c0a3580 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:29.490 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc6c1012a0 0x7fdc6c19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:29.490 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc6c0ff700 0x7fdc6c195440 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:29.490 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 --2- 172.21.15.149:0/836698089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc6c100900 0x7fdc6c195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:29.490 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 >> 172.21.15.149:0/836698089 conn(0x7fdc6c0fac80 msgr2=0x7fdc6c0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:02:29.490 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 shutdown_connections 2023-07-20T19:02:29.490 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:02:29.433+0000 7fdc72d82700 1 -- 172.21.15.149:0/836698089 wait complete. 2023-07-20T19:02:29.919 DEBUG:teuthology.orchestra.run.smithi149:osd.1> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T19:02:29.921 INFO:tasks.cephadm:Deploying osd.2 on smithi156 with /dev/vg_nvme/lv_4... 2023-07-20T19:02:29.922 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T19:02:29.952 INFO:journalctl@ceph.osd.1.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:02:30.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:30.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:29 smithi156 ceph-mon[129228]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:30.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:30.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:30.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:30.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:29 smithi061 ceph-mon[127454]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:30.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:30.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:30.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:30.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:30.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:29 smithi149 ceph-mon[128885]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:30.482 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:02:30.763 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:30 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1[133856]: 2023-07-20T19:02:30.351+0000 7f21d9764640 -1 osd.1 0 log_to_monitors true 2023-07-20T19:02:31.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:30 smithi156 ceph-mon[129228]: from='osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T19:02:31.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:30 smithi156 ceph-mon[129228]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T19:02:31.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:30 smithi149 ceph-mon[128885]: from='osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T19:02:31.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:30 smithi149 ceph-mon[128885]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T19:02:31.215 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:30 smithi061 ceph-mon[127454]: from='osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T19:02:31.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:30 smithi061 ceph-mon[127454]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T19:02:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T19:02:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T19:02:31.971 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-20T19:02:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-20T19:02:31.971 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00818598 s, 1.3 GB/s 2023-07-20T19:02:31.971 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-20T19:02:32.147 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:31 smithi156 ceph-mon[129228]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T19:02:32.148 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:31 smithi156 ceph-mon[129228]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T19:02:32.148 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:31 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:32.148 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:31 smithi156 ceph-mon[129228]: from='osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:02:32.148 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:31 smithi156 ceph-mon[129228]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:02:32.148 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:31 smithi156 ceph-mon[129228]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:32.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:31 smithi061 ceph-mon[127454]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T19:02:32.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:31 smithi061 ceph-mon[127454]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T19:02:32.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:31 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:32.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:31 smithi061 ceph-mon[127454]: from='osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:02:32.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:31 smithi061 ceph-mon[127454]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:02:32.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:31 smithi061 ceph-mon[127454]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:32.267 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1[133856]: 2023-07-20T19:02:31.807+0000 7f21ceddc700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T19:02:32.268 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1[133856]: 2023-07-20T19:02:31.811+0000 7f21c73cb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:02:32.268 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-mon[128885]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T19:02:32.268 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-mon[128885]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T19:02:32.268 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:32.268 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-mon[128885]: from='osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:02:32.268 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-mon[128885]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:02:32.269 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:31 smithi149 ceph-mon[128885]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T19:02:32.435 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2023-07-20T19:02:32.994 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:02:33.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:02:33.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T19:02:33.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:33.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:33.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi149 2023-07-20T19:02:33.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:33.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi149 to 5879M 2023-07-20T19:02:33.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:33.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:33.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:02:33.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T19:02:33.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:33.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi149 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi149 to 5879M 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:33.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:33.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:02:33.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi149 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi149 to 5879M 2023-07-20T19:02:33.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:33.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:33.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:33.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:34.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:33 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:02:34.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:33 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:02:34.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:33 smithi156 ceph-mon[129228]: osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084] boot 2023-07-20T19:02:34.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:33 smithi156 ceph-mon[129228]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T19:02:34.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:34.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:33 smithi156 ceph-mon[129228]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:34.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:33 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:02:34.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:33 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:02:34.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:33 smithi061 ceph-mon[127454]: osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084] boot 2023-07-20T19:02:34.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:33 smithi061 ceph-mon[127454]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T19:02:34.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:34.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:33 smithi061 ceph-mon[127454]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:34.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:33 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:02:34.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:33 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:02:34.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:33 smithi149 ceph-mon[128885]: osd.1 [v2:172.21.15.149:6800/414280084,v1:172.21.15.149:6801/414280084] boot 2023-07-20T19:02:34.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:33 smithi149 ceph-mon[128885]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T19:02:34.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T19:02:34.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:33 smithi149 ceph-mon[128885]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:34.418 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 -- 172.21.15.156:0/927700578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 msgr2=0x7f1c1c106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:34.418 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 --2- 172.21.15.156:0/927700578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c106050 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009990 tx=0x7f1c0c009ca0 comp rx=0 tx=0).stop 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 -- 172.21.15.156:0/927700578 shutdown_connections 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 --2- 172.21.15.156:0/927700578 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c1c106590 0x7f1c1c108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 --2- 172.21.15.156:0/927700578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c1c101330 0x7f1c1c103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 --2- 172.21.15.156:0/927700578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c106050 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 -- 172.21.15.156:0/927700578 >> 172.21.15.156:0/927700578 conn(0x7f1c1c0fac80 msgr2=0x7f1c1c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 -- 172.21.15.156:0/927700578 shutdown_connections 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 -- 172.21.15.156:0/927700578 wait complete. 2023-07-20T19:02:34.419 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.363+0000 7f1c2200a700 1 Processor -- start 2023-07-20T19:02:34.420 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 -- start start 2023-07-20T19:02:34.421 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c1c101330 0x7f1c1c1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:34.421 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c1c106590 0x7f1c1c19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c19e140 con 0x7f1c1c106590 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c19e280 con 0x7f1c1c101330 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c2200a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c19e410 con 0x7f1c1c103c60 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48380/0 (socket says 172.21.15.156:48380) 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 -- 172.21.15.156:0/3046729264 learned_addr learned my addr 172.21.15.156:0/3046729264 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:02:34.422 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1bfff700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c1c106590 0x7f1c1c19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 -- 172.21.15.156:0/3046729264 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c1c101330 msgr2=0x7f1c1c1965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1b7fe700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c1c101330 0x7f1c1c1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c1c101330 0x7f1c1c1965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 -- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c1c106590 msgr2=0x7f1c1c19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c1c106590 0x7f1c1c19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.364+0000 7f1c1affd700 1 -- 172.21.15.156:0/3046729264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0095f0 con 0x7f1c1c103c60 2023-07-20T19:02:34.423 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.365+0000 7f1c1affd700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c196af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c005910 tx=0x7f1c0c00f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:02:34.424 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.365+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c01d070 con 0x7f1c1c103c60 2023-07-20T19:02:34.424 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.365+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c0c00fc60 con 0x7f1c1c103c60 2023-07-20T19:02:34.424 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.365+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0176e0 con 0x7f1c1c103c60 2023-07-20T19:02:34.424 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.365+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c19e690 con 0x7f1c1c103c60 2023-07-20T19:02:34.424 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.365+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c19eaa0 con 0x7f1c1c103c60 2023-07-20T19:02:34.424 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.366+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c0fc800 con 0x7f1c1c103c60 2023-07-20T19:02:34.425 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.367+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f1c0c022470 con 0x7f1c1c103c60 2023-07-20T19:02:34.425 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.368+0000 7f1c18ff9700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1c040a1140 0x7f1c040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:34.426 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.368+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f1c0c0cc2e0 con 0x7f1c1c103c60 2023-07-20T19:02:34.426 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.368+0000 7f1c1b7fe700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1c040a1140 0x7f1c040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:34.426 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.368+0000 7f1c1b7fe700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1c040a1140 0x7f1c040a3600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c10006fd0 tx=0x7f1c10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:02:34.427 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.372+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1c0c097310 con 0x7f1c1c103c60 2023-07-20T19:02:34.617 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:34.562+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c1c060bb0 con 0x7f1c040a1140 2023-07-20T19:02:35.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:34 smithi156 ceph-mon[129228]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T19:02:35.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:34 smithi156 ceph-mon[129228]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:35.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:34 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:35.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:34 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:35.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:34 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:35.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:34 smithi061 ceph-mon[127454]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T19:02:35.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:34 smithi061 ceph-mon[127454]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:35.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:34 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:35.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:34 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:35.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:34 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:35.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:34 smithi149 ceph-mon[128885]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T19:02:35.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:34 smithi149 ceph-mon[128885]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:35.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:34 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:35.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:34 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:35.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:34 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:36.027 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:35 smithi156 ceph-mon[129228]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:36.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:35 smithi061 ceph-mon[127454]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:36.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:35 smithi149 ceph-mon[128885]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:37.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:37 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/3878817885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]: dispatch 2023-07-20T19:02:37.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:37 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]: dispatch 2023-07-20T19:02:37.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:37 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]': finished 2023-07-20T19:02:37.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:37 smithi156 ceph-mon[129228]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T19:02:37.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:37.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:37 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/3878817885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]: dispatch 2023-07-20T19:02:37.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:37 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]: dispatch 2023-07-20T19:02:37.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:37 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]': finished 2023-07-20T19:02:37.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:37 smithi061 ceph-mon[127454]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T19:02:37.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:37.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:37 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/3878817885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]: dispatch 2023-07-20T19:02:37.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:37 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]: dispatch 2023-07-20T19:02:37.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:37 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34a03029-87a9-49e1-b262-32be5070df0e"}]': finished 2023-07-20T19:02:37.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:37 smithi149 ceph-mon[128885]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T19:02:37.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:38.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:38 smithi149 ceph-mon[128885]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:38.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:38 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/144963870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:38.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:38 smithi156 ceph-mon[129228]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:38.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:38 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/144963870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:38 smithi061 ceph-mon[127454]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:38.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:38 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/144963870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:40.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:39 smithi149 ceph-mon[128885]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:40.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:39 smithi156 ceph-mon[129228]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:40.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:39 smithi061 ceph-mon[127454]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:41.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:41 smithi156 ceph-mon[129228]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:42.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:41 smithi149 ceph-mon[128885]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:42.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:41 smithi061 ceph-mon[127454]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:43.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T19:02:43.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:43.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T19:02:43.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:44.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T19:02:44.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:44.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:44 smithi156 ceph-mon[129228]: Deploying daemon osd.2 on smithi156 2023-07-20T19:02:44.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:44 smithi156 ceph-mon[129228]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:44.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:44 smithi061 ceph-mon[127454]: Deploying daemon osd.2 on smithi156 2023-07-20T19:02:44.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:44 smithi061 ceph-mon[127454]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:45.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:44 smithi149 ceph-mon[128885]: Deploying daemon osd.2 on smithi156 2023-07-20T19:02:45.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:44 smithi149 ceph-mon[128885]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:46.012 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:45 smithi156 ceph-mon[129228]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:46.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:45 smithi149 ceph-mon[128885]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:46.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:45 smithi061 ceph-mon[127454]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:48.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:48 smithi149 ceph-mon[128885]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:48.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:48.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:48.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:48 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:48.832 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:48 smithi156 ceph-mon[129228]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:48.832 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:48.832 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:48.832 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:48 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:48.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:48 smithi061 ceph-mon[127454]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:48.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:48.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:48.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:48 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:50.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:49 smithi149 ceph-mon[128885]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:50.159 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:49 smithi156 ceph-mon[129228]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:50.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:49 smithi061 ceph-mon[127454]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:50.532 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.477+0000 7f1c18ff9700 1 -- 172.21.15.156:0/3046729264 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1c1c060bb0 con 0x7f1c040a1140 2023-07-20T19:02:50.532 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 2 on host 'smithi156' 2023-07-20T19:02:50.534 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1c040a1140 msgr2=0x7f1c040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:50.535 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1c040a1140 0x7f1c040a3600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c10006fd0 tx=0x7f1c10008040 comp rx=0 tx=0).stop 2023-07-20T19:02:50.535 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 msgr2=0x7f1c1c196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:50.535 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c196af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c005910 tx=0x7f1c0c00f7b0 comp rx=0 tx=0).stop 2023-07-20T19:02:50.535 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 shutdown_connections 2023-07-20T19:02:50.535 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1c040a1140 0x7f1c040a3600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:50.535 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c1c106590 0x7f1c1c19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:50.536 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c1c101330 0x7f1c1c1965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:50.536 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 --2- 172.21.15.156:0/3046729264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c1c103c60 0x7f1c1c196af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:50.536 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.479+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 >> 172.21.15.156:0/3046729264 conn(0x7f1c1c0fac80 msgr2=0x7f1c1c101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:02:50.536 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.480+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 shutdown_connections 2023-07-20T19:02:50.536 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:02:50.480+0000 7f1c2200a700 1 -- 172.21.15.156:0/3046729264 wait complete. 2023-07-20T19:02:50.869 DEBUG:teuthology.orchestra.run.smithi156:osd.2> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T19:02:50.871 INFO:tasks.cephadm:Deploying osd.3 on smithi061 with /dev/vg_nvme/lv_3... 2023-07-20T19:02:50.871 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T19:02:50.901 INFO:journalctl@ceph.osd.2.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:02:51.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:51.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:51.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:50 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:51.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:51.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:51.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:50 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:51.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:51.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:51.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:51.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:50 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:02:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:02:51.953 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2[135068]: 2023-07-20T19:02:51.727+0000 7efe86d54640 -1 osd.2 0 log_to_monitors true 2023-07-20T19:02:52.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:51 smithi061 ceph-mon[127454]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:52.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:51 smithi061 ceph-mon[127454]: from='osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T19:02:52.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:51 smithi061 ceph-mon[127454]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T19:02:52.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:51 smithi149 ceph-mon[128885]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:52.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:51 smithi149 ceph-mon[128885]: from='osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T19:02:52.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:51 smithi149 ceph-mon[128885]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T19:02:52.413 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:51 smithi156 ceph-mon[129228]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:52.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:51 smithi156 ceph-mon[129228]: from='osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T19:02:52.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:51 smithi156 ceph-mon[129228]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T19:02:52.985 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T19:02:52.986 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T19:02:52.986 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 10+0 records in 2023-07-20T19:02:52.986 INFO:teuthology.orchestra.run.smithi061.stderr:10+0 records out 2023-07-20T19:02:52.986 INFO:teuthology.orchestra.run.smithi061.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127354 s, 823 MB/s 2023-07-20T19:02:52.986 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping successful for: 2023-07-20T19:02:53.163 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2[135068]: 2023-07-20T19:02:52.912+0000 7efe7c3cc700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T19:02:53.164 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2[135068]: 2023-07-20T19:02:52.916+0000 7efe769bf700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:02:53.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-mon[129228]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T19:02:53.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-mon[129228]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T19:02:53.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:53.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-mon[129228]: from='osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:02:53.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:52 smithi156 ceph-mon[129228]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:02:53.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:52 smithi061 ceph-mon[127454]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T19:02:53.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:52 smithi061 ceph-mon[127454]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T19:02:53.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:52 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:53.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:52 smithi061 ceph-mon[127454]: from='osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:02:53.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:52 smithi061 ceph-mon[127454]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:02:53.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:52 smithi149 ceph-mon[128885]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T19:02:53.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:52 smithi149 ceph-mon[128885]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T19:02:53.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:52 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:53.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:52 smithi149 ceph-mon[128885]: from='osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:02:53.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:52 smithi149 ceph-mon[128885]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:02:53.413 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi061:vg_nvme/lv_3 2023-07-20T19:02:53.974 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:02:54.227 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:02:54.227 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T19:02:54.227 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:54.227 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:54.227 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi156 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi156 to 17168M 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:54.228 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.229 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:53 smithi061 ceph-mon[127454]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:54.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:02:54.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T19:02:54.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:54.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:54.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi156 2023-07-20T19:02:54.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi156 to 17168M 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.302 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:53 smithi149 ceph-mon[128885]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:54.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:02:54.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T19:02:54.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:54.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:54.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi156 2023-07-20T19:02:54.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi156 to 17168M 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:02:54.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:54.416 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:53 smithi156 ceph-mon[129228]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T19:02:55.156 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 -- 172.21.15.61:0/2858235878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c4101af0 msgr2=0x7f79c4101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:55.156 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/2858235878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c4101af0 0x7f79c4101f50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009990 tx=0x7f79b4009ca0 comp rx=0 tx=0).stop 2023-07-20T19:02:55.157 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 -- 172.21.15.61:0/2858235878 shutdown_connections 2023-07-20T19:02:55.157 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/2858235878 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4102490 0x7f79c4108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:55.157 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/2858235878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79c40feb10 0x7f79c40fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:55.157 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/2858235878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c4101af0 0x7f79c4101f50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:55.158 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 -- 172.21.15.61:0/2858235878 >> 172.21.15.61:0/2858235878 conn(0x7f79c40fa450 msgr2=0x7f79c40fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:02:55.158 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 -- 172.21.15.61:0/2858235878 shutdown_connections 2023-07-20T19:02:55.158 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.100+0000 7f79c8c6f700 1 -- 172.21.15.61:0/2858235878 wait complete. 2023-07-20T19:02:55.160 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 Processor -- start 2023-07-20T19:02:55.160 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 -- start start 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 0x7f79c4198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4101af0 0x7f79c4198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79c4102490 0x7f79c419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c41a01f0 con 0x7f79c40feb10 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c41a0330 con 0x7f79c4101af0 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c8c6f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c41a04c0 con 0x7f79c4102490 2023-07-20T19:02:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 0x7f79c4198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 0x7f79c4198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57648/0 (socket says 172.21.15.61:57648) 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c259c700 1 -- 172.21.15.61:0/941306117 learned_addr learned my addr 172.21.15.61:0/941306117 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c2d9d700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79c4102490 0x7f79c419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.101+0000 7f79c1d9b700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4101af0 0x7f79c4198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c259c700 1 -- 172.21.15.61:0/941306117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79c4102490 msgr2=0x7f79c419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c259c700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79c4102490 0x7f79c419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c259c700 1 -- 172.21.15.61:0/941306117 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4101af0 msgr2=0x7f79c4198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:02:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c259c700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4101af0 0x7f79c4198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c259c700 1 -- 172.21.15.61:0/941306117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b40095f0 con 0x7f79c40feb10 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c1d9b700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4101af0 0x7f79c4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c259c700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 0x7f79c4198660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79b800d9b0 tx=0x7f79b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8004020 con 0x7f79c40feb10 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79b8009730 con 0x7f79c40feb10 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c41a07a0 con 0x7f79c40feb10 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c41a0ca0 con 0x7f79c40feb10 2023-07-20T19:02:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.102+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8003a30 con 0x7f79c40feb10 2023-07-20T19:02:55.164 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.103+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c410c040 con 0x7f79c40feb10 2023-07-20T19:02:55.164 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.104+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f79b80098a0 con 0x7f79c40feb10 2023-07-20T19:02:55.164 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.104+0000 7f79b37fe700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f79ac0a10c0 0x7f79ac0a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:02:55.164 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.105+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f79b80cb310 con 0x7f79c40feb10 2023-07-20T19:02:55.164 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.105+0000 7f79c1d9b700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f79ac0a10c0 0x7f79ac0a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:02:55.165 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.105+0000 7f79c1d9b700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f79ac0a10c0 0x7f79ac0a3580 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79b400b5c0 tx=0x7f79b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:02:55.165 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.108+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f79b8096090 con 0x7f79c40feb10 2023-07-20T19:02:55.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:54 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:02:55.216 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:54 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:02:55.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:55.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:54 smithi061 ceph-mon[127454]: osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812] boot 2023-07-20T19:02:55.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:54 smithi061 ceph-mon[127454]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T19:02:55.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:55.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:54 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:02:55.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:54 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:02:55.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:55.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:54 smithi149 ceph-mon[128885]: osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812] boot 2023-07-20T19:02:55.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:54 smithi149 ceph-mon[128885]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T19:02:55.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:55.314+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f79c41a1390 con 0x7f79ac0a10c0 2023-07-20T19:02:55.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:54 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:02:55.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:54 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:02:55.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:55.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:54 smithi156 ceph-mon[129228]: osd.2 [v2:172.21.15.156:6800/738107812,v1:172.21.15.156:6801/738107812] boot 2023-07-20T19:02:55.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:54 smithi156 ceph-mon[129228]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T19:02:55.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T19:02:56.439 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:56 smithi061 ceph-mon[127454]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T19:02:56.439 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:56 smithi061 ceph-mon[127454]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:56.439 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:56.439 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:56.439 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:56.440 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:56 smithi061 ceph-mon[127454]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:02:56.550 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:56 smithi149 ceph-mon[128885]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T19:02:56.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:56 smithi149 ceph-mon[128885]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:56.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:56.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:56.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:56.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:56 smithi149 ceph-mon[128885]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:02:56.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:56 smithi156 ceph-mon[129228]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T19:02:56.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:56 smithi156 ceph-mon[129228]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:02:56.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:02:56.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:02:56.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:02:56.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:56 smithi156 ceph-mon[129228]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:02:57.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T19:02:57.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T19:02:57.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:57 smithi061 ceph-mon[127454]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T19:02:57.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T19:02:57.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T19:02:57.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T19:02:57.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:57 smithi149 ceph-mon[128885]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T19:02:57.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T19:02:57.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T19:02:57.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T19:02:57.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:57 smithi156 ceph-mon[129228]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T19:02:57.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T19:02:58.216 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:57 smithi061 sudo[143509]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T19:02:58.216 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:57 smithi061 sudo[143509]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:02:58.217 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:57 smithi061 sudo[143509]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:02:58.217 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:57 smithi061 sudo[143509]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:02:58.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T19:02:58.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T19:02:58.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/171464376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5711133-530f-488a-9db4-2b8a61b7ea02"}]: dispatch 2023-07-20T19:02:58.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/171464376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5711133-530f-488a-9db4-2b8a61b7ea02"}]': finished 2023-07-20T19:02:58.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T19:02:58.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:02:58.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:58 smithi149 ceph-mon[128885]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:02:58.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T19:02:58.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T19:02:58.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/171464376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5711133-530f-488a-9db4-2b8a61b7ea02"}]: dispatch 2023-07-20T19:02:58.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/171464376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5711133-530f-488a-9db4-2b8a61b7ea02"}]': finished 2023-07-20T19:02:58.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T19:02:58.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:02:58.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:58 smithi156 ceph-mon[129228]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:02:58.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T19:02:58.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T19:02:58.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/171464376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5711133-530f-488a-9db4-2b8a61b7ea02"}]: dispatch 2023-07-20T19:02:58.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/171464376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5711133-530f-488a-9db4-2b8a61b7ea02"}]': finished 2023-07-20T19:02:58.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T19:02:58.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:02:58.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:58 smithi061 ceph-mon[127454]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:02:58.718 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:58 smithi061 sudo[143541]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T19:02:58.718 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:58 smithi061 sudo[143541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:02:58.718 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:58 smithi061 sudo[143541]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:02:58.718 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:02:58 smithi061 sudo[143541]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:02:58.772 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136152]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T19:02:58.772 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136152]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:02:58.772 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136152]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:02:58.773 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136152]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:02:59.050 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136155]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T19:02:59.051 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:02:59.051 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136155]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:02:59.051 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:02:58 smithi149 sudo[136155]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:02:59.288 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:02:59.233+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f79b8009b80 con 0x7f79c40feb10 2023-07-20T19:02:59.295 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:59.295 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:59 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1995388849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:59.295 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:02:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:59.296 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:58 smithi156 sudo[137313]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T19:02:59.296 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:58 smithi156 sudo[137313]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:02:59.296 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:58 smithi156 sudo[137313]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:02:59.296 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:59 smithi156 sudo[137313]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:02:59.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:59.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:59 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1995388849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:59.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:02:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:59.664 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:59 smithi156 sudo[137316]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T19:02:59.664 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:59 smithi156 sudo[137316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:02:59.665 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:59 smithi156 sudo[137316]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:02:59.665 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:02:59 smithi156 sudo[137316]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:02:59.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:02:59.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1995388849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:02:59.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:00.212 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 sudo[144149]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T19:03:00.212 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 sudo[144149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:03:00.212 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 sudo[144149]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:03:00.212 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:02:59 smithi061 sudo[144149]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:03:00.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 sudo[144336]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T19:03:00.466 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 sudo[144336]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:03:00.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 sudo[144336]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:03:00.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 sudo[144336]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:03:00.705 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137319]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T19:03:00.706 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137319]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:03:00.706 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137319]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:03:00.706 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137319]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:03:00.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T19:03:00.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:00.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:00.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:00.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:00.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:00.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:00.968 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:00.968 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:00.968 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:00.969 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:00.969 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 sudo[136158]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T19:03:00.969 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 sudo[136158]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:03:00.969 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 sudo[136158]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:03:00.970 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:00 smithi149 sudo[136158]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:03:00.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T19:03:00.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:00.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:00.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:00.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:00.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:00.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:00.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:00.973 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:00.973 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:00.973 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:01.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137322]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T19:03:01.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137322]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:03:01.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137322]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:03:01.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 sudo[137322]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:03:01.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T19:03:01.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:01.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:01.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:01.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:01.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:01.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:01.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:01.171 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:01.171 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:01.171 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:01.300 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 sudo[136161]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T19:03:01.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 sudo[136161]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T19:03:01.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 sudo[136161]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T19:03:01.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 sudo[136161]: pam_unix(sudo:session): session closed for user root 2023-07-20T19:03:01.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:01.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:01.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:01.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:01.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:01.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:01 smithi061 ceph-mon[127454]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:02.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:02.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:02.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:02.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:02.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:02.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:02.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:02.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:01 smithi149 ceph-mon[128885]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:02.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:02.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:02.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T19:03:02.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T19:03:02.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T19:03:02.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T19:03:02.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T19:03:02.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:01 smithi156 ceph-mon[129228]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:04.025 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:03 smithi061 ceph-mon[127454]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-20T19:03:04.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:03 smithi149 ceph-mon[128885]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-20T19:03:04.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:03 smithi156 ceph-mon[129228]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-20T19:03:04.937 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:04 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T19:03:04.937 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:04 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:04.938 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:04 smithi061 ceph-mon[127454]: Deploying daemon osd.3 on smithi061 2023-07-20T19:03:05.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:04 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T19:03:05.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:04 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:05.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:04 smithi149 ceph-mon[128885]: Deploying daemon osd.3 on smithi061 2023-07-20T19:03:05.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:04 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T19:03:05.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:04 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:05.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:04 smithi156 ceph-mon[129228]: Deploying daemon osd.3 on smithi061 2023-07-20T19:03:06.050 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:05 smithi149 ceph-mon[128885]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-20T19:03:06.092 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:05 smithi061 ceph-mon[127454]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-20T19:03:06.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:05 smithi156 ceph-mon[129228]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-20T19:03:08.716 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:08 smithi061 ceph-mon[127454]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-20T19:03:08.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:08 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:08.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:08 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:08.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:08 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:08.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:08 smithi149 ceph-mon[128885]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-20T19:03:08.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:08 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:08.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:08 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:08.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:08 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:08.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:08 smithi156 ceph-mon[129228]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-20T19:03:08.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:08 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:08.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:08 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:08.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:08 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:09.917 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:09 smithi061 ceph-mon[127454]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-20T19:03:10.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:09 smithi149 ceph-mon[128885]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-20T19:03:10.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:09 smithi156 ceph-mon[129228]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-20T19:03:11.072 INFO:teuthology.orchestra.run.smithi061.stdout:Created osd(s) 3 on host 'smithi061' 2023-07-20T19:03:11.073 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.017+0000 7f79b37fe700 1 -- 172.21.15.61:0/941306117 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f79c41a1390 con 0x7f79ac0a10c0 2023-07-20T19:03:11.074 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f79ac0a10c0 msgr2=0x7f79ac0a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f79ac0a10c0 0x7f79ac0a3580 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79b400b5c0 tx=0x7f79b4005fb0 comp rx=0 tx=0).stop 2023-07-20T19:03:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 msgr2=0x7f79c4198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 0x7f79c4198660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79b800d9b0 tx=0x7f79b800dcc0 comp rx=0 tx=0).stop 2023-07-20T19:03:11.075 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 shutdown_connections 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f79ac0a10c0 0x7f79ac0a3580 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79c4102490 0x7f79c419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79c40feb10 0x7f79c4198660 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 --2- 172.21.15.61:0/941306117 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f79c4101af0 0x7f79c4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 >> 172.21.15.61:0/941306117 conn(0x7f79c40fa450 msgr2=0x7f79c4107330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 shutdown_connections 2023-07-20T19:03:11.076 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:11.020+0000 7f79c8c6f700 1 -- 172.21.15.61:0/941306117 wait complete. 2023-07-20T19:03:11.469 DEBUG:teuthology.orchestra.run.smithi061:osd.3> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T19:03:11.472 INFO:tasks.cephadm:Deploying osd.4 on smithi149 with /dev/vg_nvme/lv_3... 2023-07-20T19:03:11.473 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T19:03:11.502 INFO:journalctl@ceph.osd.3.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:03:11.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:11.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:11.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:11.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:11.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:11.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:11.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:12.040 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:03:12.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:12 smithi149 ceph-mon[128885]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-20T19:03:12.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:12 smithi149 ceph-mon[128885]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T19:03:12.693 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:12 smithi061 ceph-mon[127454]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-20T19:03:12.694 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:12 smithi061 ceph-mon[127454]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T19:03:12.694 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:03:12 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:03:12.255+0000 7fb1ad73c640 -1 osd.3 0 log_to_monitors true 2023-07-20T19:03:12.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:12 smithi156 ceph-mon[129228]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-20T19:03:12.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:12 smithi156 ceph-mon[129228]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T19:03:13.502 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:03:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:03:13.436+0000 7fb1a2db4700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T19:03:13.502 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:03:13 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:03:13.441+0000 7fb19cba6700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:03:13.502 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:13 smithi061 ceph-mon[127454]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T19:03:13.503 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:13 smithi061 ceph-mon[127454]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T19:03:13.503 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:13.503 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:13 smithi061 ceph-mon[127454]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:03:13.656 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T19:03:13.657 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T19:03:13.657 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2023-07-20T19:03:13.657 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2023-07-20T19:03:13.657 INFO:teuthology.orchestra.run.smithi149.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110378 s, 950 MB/s 2023-07-20T19:03:13.658 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2023-07-20T19:03:13.799 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:13 smithi149 ceph-mon[128885]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T19:03:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:13 smithi149 ceph-mon[128885]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T19:03:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:13.800 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:13 smithi149 ceph-mon[128885]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:03:13.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:13 smithi156 ceph-mon[129228]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T19:03:13.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:13 smithi156 ceph-mon[129228]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T19:03:13.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:13.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:13 smithi156 ceph-mon[129228]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:03:14.086 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi149:vg_nvme/lv_3 2023-07-20T19:03:14.637 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:03:14.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:03:14.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T19:03:14.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi061 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:14.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi061 to 8584M 2023-07-20T19:03:14.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:14.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:14.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:14 smithi149 ceph-mon[128885]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:14.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:03:14.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T19:03:14.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi061 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:14.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi061 to 8584M 2023-07-20T19:03:14.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:14.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:14.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:14 smithi156 ceph-mon[129228]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:14.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:03:14.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T19:03:14.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:14.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:14.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi061 2023-07-20T19:03:14.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi061 to 8584M 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:14.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:14.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:14.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:14 smithi061 ceph-mon[127454]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T19:03:15.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:15 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:03:15.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:15 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:03:15.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:15 smithi149 ceph-mon[128885]: osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823] boot 2023-07-20T19:03:15.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:15 smithi149 ceph-mon[128885]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T19:03:15.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:15.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:15 smithi149 ceph-mon[128885]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T19:03:15.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 -- 172.21.15.149:0/3277972129 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c101a00 msgr2=0x7ffb3c101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:15.885 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 --2- 172.21.15.149:0/3277972129 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c101a00 0x7ffb3c101e60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0099c0 tx=0x7ffb2c009cd0 comp rx=0 tx=0).stop 2023-07-20T19:03:15.885 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 -- 172.21.15.149:0/3277972129 shutdown_connections 2023-07-20T19:03:15.885 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 --2- 172.21.15.149:0/3277972129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:15.886 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 --2- 172.21.15.149:0/3277972129 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb3c100800 0x7ffb3c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:15.887 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 --2- 172.21.15.149:0/3277972129 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c101a00 0x7ffb3c101e60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:15.887 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 -- 172.21.15.149:0/3277972129 >> 172.21.15.149:0/3277972129 conn(0x7ffb3c0fbd60 msgr2=0x7ffb3c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:03:15.887 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 -- 172.21.15.149:0/3277972129 shutdown_connections 2023-07-20T19:03:15.887 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.825+0000 7ffb416f2700 1 -- 172.21.15.149:0/3277972129 wait complete. 2023-07-20T19:03:15.888 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 Processor -- start 2023-07-20T19:03:15.888 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 -- start start 2023-07-20T19:03:15.888 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb3c100800 0x7ffb3c196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb3c101a00 0x7ffb3c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c19dfb0 con 0x7ffb3c100800 2023-07-20T19:03:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c19e0f0 con 0x7ffb3c1023a0 2023-07-20T19:03:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb416f2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c19e280 con 0x7ffb3c101a00 2023-07-20T19:03:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c19d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:44132/0 (socket says 172.21.15.149:44132) 2023-07-20T19:03:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 -- 172.21.15.149:0/444174234 learned_addr learned my addr 172.21.15.149:0/444174234 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-07-20T19:03:15.891 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3affd700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb3c100800 0x7ffb3c196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:15.891 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3a7fc700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb3c101a00 0x7ffb3c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:15.891 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 -- 172.21.15.149:0/444174234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb3c101a00 msgr2=0x7ffb3c196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:15.892 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb3c101a00 0x7ffb3c196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:15.892 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 -- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb3c100800 msgr2=0x7ffb3c196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:15.892 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb3c100800 0x7ffb3c196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:15.893 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.826+0000 7ffb3b7fe700 1 -- 172.21.15.149:0/444174234 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c0095f0 con 0x7ffb3c1023a0 2023-07-20T19:03:15.893 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.827+0000 7ffb3b7fe700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c19d9e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb34007d70 tx=0x7ffb3400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:03:15.893 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.827+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mon.2 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3400ad40 con 0x7ffb3c1023a0 2023-07-20T19:03:15.893 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.827+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3c19e560 con 0x7ffb3c1023a0 2023-07-20T19:03:15.894 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.827+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mon.2 v2:172.21.15.149:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb340128e0 con 0x7ffb3c1023a0 2023-07-20T19:03:15.894 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.827+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mon.2 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34011780 con 0x7ffb3c1023a0 2023-07-20T19:03:15.894 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.827+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3c19ea10 con 0x7ffb3c1023a0 2023-07-20T19:03:15.895 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.828+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3c107db0 con 0x7ffb3c1023a0 2023-07-20T19:03:15.896 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.829+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mon.2 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffb34012a50 con 0x7ffb3c1023a0 2023-07-20T19:03:15.896 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.829+0000 7ffb23fff700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb240a1150 0x7ffb240a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:15.897 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.830+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mon.2 v2:172.21.15.149:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7ffb340cbf10 con 0x7ffb3c1023a0 2023-07-20T19:03:15.897 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.830+0000 7ffb3affd700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb240a1150 0x7ffb240a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:15.897 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.830+0000 7ffb3affd700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb240a1150 0x7ffb240a3610 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c101860 tx=0x7ffb30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:03:15.898 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:15.833+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mon.2 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffb340967c0 con 0x7ffb3c1023a0 2023-07-20T19:03:15.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:15 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:03:15.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:15 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:03:15.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:15 smithi156 ceph-mon[129228]: osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823] boot 2023-07-20T19:03:15.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:15 smithi156 ceph-mon[129228]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T19:03:15.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:15.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:15 smithi156 ceph-mon[129228]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T19:03:15.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:15 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:03:15.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:15 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:03:15.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:15 smithi061 ceph-mon[127454]: osd.3 [v2:172.21.15.61:6810/1371683823,v1:172.21.15.61:6811/1371683823] boot 2023-07-20T19:03:15.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:15 smithi061 ceph-mon[127454]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T19:03:15.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T19:03:15.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:15 smithi061 ceph-mon[127454]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T19:03:16.081 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:16.024+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb3c060bb0 con 0x7ffb240a1150 2023-07-20T19:03:16.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:16 smithi149 ceph-mon[128885]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:16.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:16 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:16.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:16 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:16.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:16 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:16.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:16 smithi156 ceph-mon[129228]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:16.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:16 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:16.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:16 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:16.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:16 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:16.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:16 smithi061 ceph-mon[127454]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:16.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:16 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:16.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:16 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:16.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:16 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:17.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:17 smithi149 ceph-mon[128885]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:17.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:17 smithi149 ceph-mon[128885]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T19:03:17.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:17 smithi156 ceph-mon[129228]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:17.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:17 smithi156 ceph-mon[129228]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T19:03:17.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:17 smithi061 ceph-mon[127454]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:17.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:17 smithi061 ceph-mon[127454]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T19:03:18.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T19:03:18.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:18.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/2480672461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]: dispatch 2023-07-20T19:03:18.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]: dispatch 2023-07-20T19:03:18.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]': finished 2023-07-20T19:03:18.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T19:03:18.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:18 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:18.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T19:03:18.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:18.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/2480672461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]: dispatch 2023-07-20T19:03:18.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]: dispatch 2023-07-20T19:03:18.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]': finished 2023-07-20T19:03:18.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T19:03:18.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:18 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:18.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T19:03:18.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:18.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/2480672461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]: dispatch 2023-07-20T19:03:18.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]: dispatch 2023-07-20T19:03:18.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6062cdd3-5bf1-458a-8454-253627db736f"}]': finished 2023-07-20T19:03:18.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T19:03:18.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:18 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:19.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:19 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/2620933142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:03:19.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:19 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/2620933142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:03:19.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:19 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/2620933142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:03:20.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:20 smithi149 ceph-mon[128885]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-07-20T19:03:20.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:20 smithi156 ceph-mon[129228]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-07-20T19:03:20.966 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:20 smithi061 ceph-mon[127454]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-07-20T19:03:22.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:21 smithi149 ceph-mon[128885]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:03:22.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:21 smithi156 ceph-mon[129228]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:03:22.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:21 smithi061 ceph-mon[127454]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:03:24.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:23 smithi149 ceph-mon[128885]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-07-20T19:03:24.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:23 smithi156 ceph-mon[129228]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-07-20T19:03:24.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:23 smithi061 ceph-mon[127454]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-07-20T19:03:25.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:25 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T19:03:25.301 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:25 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T19:03:25.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:25.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:25 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T19:03:25.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:25 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:26.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:26 smithi156 ceph-mon[129228]: Deploying daemon osd.4 on smithi149 2023-07-20T19:03:26.414 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:26 smithi156 ceph-mon[129228]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-20T19:03:26.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:26 smithi149 ceph-mon[128885]: Deploying daemon osd.4 on smithi149 2023-07-20T19:03:26.439 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:26 smithi149 ceph-mon[128885]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-20T19:03:26.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:26 smithi061 ceph-mon[127454]: Deploying daemon osd.4 on smithi149 2023-07-20T19:03:26.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:26 smithi061 ceph-mon[127454]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-20T19:03:28.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:28 smithi149 ceph-mon[128885]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:03:28.914 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:28 smithi156 ceph-mon[129228]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:03:28.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:28 smithi061 ceph-mon[127454]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:03:29.859 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:29 smithi149 ceph-mon[128885]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:30.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:29 smithi156 ceph-mon[129228]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:30.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:29 smithi061 ceph-mon[127454]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:31.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:31.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:31.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:31.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:31.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:31.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:31.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:31.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:31.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:31.964 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:31 smithi149 ceph-mon[128885]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:32.164 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:31 smithi156 ceph-mon[129228]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:32.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:31 smithi061 ceph-mon[127454]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:32.902 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.845+0000 7ffb23fff700 1 -- 172.21.15.149:0/444174234 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffb3c060bb0 con 0x7ffb240a1150 2023-07-20T19:03:32.903 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 4 on host 'smithi149' 2023-07-20T19:03:32.905 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb240a1150 msgr2=0x7ffb240a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:32.905 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb240a1150 0x7ffb240a3610 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c101860 tx=0x7ffb30009260 comp rx=0 tx=0).stop 2023-07-20T19:03:32.905 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 msgr2=0x7ffb3c19d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:32.905 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c19d9e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb34007d70 tx=0x7ffb3400a340 comp rx=0 tx=0).stop 2023-07-20T19:03:32.905 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 shutdown_connections 2023-07-20T19:03:32.905 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffb240a1150 0x7ffb240a3610 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:32.906 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb3c1023a0 0x7ffb3c19d9e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:32.906 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb3c100800 0x7ffb3c196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:32.906 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 --2- 172.21.15.149:0/444174234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb3c101a00 0x7ffb3c196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:32.906 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 >> 172.21.15.149:0/444174234 conn(0x7ffb3c0fbd60 msgr2=0x7ffb3c104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:03:32.906 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 shutdown_connections 2023-07-20T19:03:32.906 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:03:32.848+0000 7ffb416f2700 1 -- 172.21.15.149:0/444174234 wait complete. 2023-07-20T19:03:33.331 DEBUG:teuthology.orchestra.run.smithi149:osd.4> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T19:03:33.333 INFO:tasks.cephadm:Deploying osd.5 on smithi156 with /dev/vg_nvme/lv_3... 2023-07-20T19:03:33.334 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T19:03:33.364 INFO:journalctl@ceph.osd.4.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:03:33.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:33.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:33.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:33.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:33.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:33.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:33.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:33.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:33.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:33.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:33.893 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:03:34.181 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:03:33 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4[139303]: 2023-07-20T19:03:33.788+0000 7fadbbe72640 -1 osd.4 0 log_to_monitors true 2023-07-20T19:03:34.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:34 smithi156 ceph-mon[129228]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:34.389 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:34 smithi156 ceph-mon[129228]: from='osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T19:03:34.389 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:34 smithi156 ceph-mon[129228]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T19:03:34.432 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:34 smithi149 ceph-mon[128885]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:34.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:34 smithi149 ceph-mon[128885]: from='osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T19:03:34.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:34 smithi149 ceph-mon[128885]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T19:03:34.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:34 smithi061 ceph-mon[127454]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:34.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:34 smithi061 ceph-mon[127454]: from='osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T19:03:34.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:34 smithi061 ceph-mon[127454]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T19:03:35.340 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T19:03:35.340 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T19:03:35.341 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-20T19:03:35.341 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-20T19:03:35.341 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00833158 s, 1.3 GB/s 2023-07-20T19:03:35.341 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-20T19:03:35.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:35 smithi061 ceph-mon[127454]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T19:03:35.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:35 smithi061 ceph-mon[127454]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T19:03:35.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:35.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:35 smithi061 ceph-mon[127454]: from='osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:03:35.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:35 smithi061 ceph-mon[127454]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:03:35.538 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:35 smithi156 ceph-mon[129228]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T19:03:35.538 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:35 smithi156 ceph-mon[129228]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T19:03:35.539 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:35.539 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:35 smithi156 ceph-mon[129228]: from='osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:03:35.539 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:35 smithi156 ceph-mon[129228]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:03:35.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-mon[128885]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T19:03:35.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-mon[128885]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T19:03:35.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:35.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-mon[128885]: from='osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:03:35.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-mon[128885]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:03:35.553 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4[139303]: 2023-07-20T19:03:35.130+0000 7fadb14ea700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T19:03:35.553 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:03:35 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4[139303]: 2023-07-20T19:03:35.135+0000 7fadabadd700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:03:35.735 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2023-07-20T19:03:36.295 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:03:36.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:03:36.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T19:03:36.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:36.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:36.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi149 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi149 to 2939M 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:36.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:36.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:36 smithi061 ceph-mon[127454]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:36.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:03:36.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T19:03:36.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:36.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi149 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi149 to 2939M 2023-07-20T19:03:36.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:36.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:36.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:36 smithi156 ceph-mon[129228]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:36.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:03:36.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T19:03:36.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:36.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:36.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi149 2023-07-20T19:03:36.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi149 to 2939M 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:36.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:36.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:36 smithi149 ceph-mon[128885]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T19:03:37.441 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.384+0000 7f452e6ba700 1 -- 172.21.15.156:0/3236856873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4528100900 msgr2=0x7f4528100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:37.441 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.384+0000 7f452e6ba700 1 --2- 172.21.15.156:0/3236856873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4528100900 0x7f4528100d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f451c009990 tx=0x7f451c009ca0 comp rx=0 tx=0).stop 2023-07-20T19:03:37.441 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 -- 172.21.15.156:0/3236856873 shutdown_connections 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 --2- 172.21.15.156:0/3236856873 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45281012a0 0x7f4528107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 --2- 172.21.15.156:0/3236856873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45280ff700 0x7f45280ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 --2- 172.21.15.156:0/3236856873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4528100900 0x7f4528100d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 -- 172.21.15.156:0/3236856873 >> 172.21.15.156:0/3236856873 conn(0x7f45280fac80 msgr2=0x7f45280fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 -- 172.21.15.156:0/3236856873 shutdown_connections 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 -- 172.21.15.156:0/3236856873 wait complete. 2023-07-20T19:03:37.442 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 Processor -- start 2023-07-20T19:03:37.443 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 -- start start 2023-07-20T19:03:37.444 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.385+0000 7f452e6ba700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 0x7f4528069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:37.445 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4528100900 0x7f452819b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:37.445 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45281012a0 0x7f452819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:37.445 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452819df50 con 0x7f4528100900 2023-07-20T19:03:37.445 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452819e0c0 con 0x7f45281012a0 2023-07-20T19:03:37.445 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452819e280 con 0x7f45280ff700 2023-07-20T19:03:37.445 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 0x7f4528069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 0x7f4528069620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48460/0 (socket says 172.21.15.156:48460) 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 -- 172.21.15.156:0/2481539298 learned_addr learned my addr 172.21.15.156:0/2481539298 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f45277fe700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4528100900 0x7f452819b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452cc57700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45281012a0 0x7f452819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 -- 172.21.15.156:0/2481539298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45281012a0 msgr2=0x7f452819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45281012a0 0x7f452819d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:37.446 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 -- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4528100900 msgr2=0x7f452819b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4528100900 0x7f452819b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 -- 172.21.15.156:0/2481539298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0095f0 con 0x7f45280ff700 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f4527fff700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 0x7f4528069620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f451800e970 tx=0x7f451800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518009790 con 0x7f45280ff700 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45180044e0 con 0x7f45280ff700 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452819e560 con 0x7f45280ff700 2023-07-20T19:03:37.447 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518003e30 con 0x7f45280ff700 2023-07-20T19:03:37.448 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.386+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452819eab0 con 0x7f45280ff700 2023-07-20T19:03:37.448 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.387+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4528106cb0 con 0x7f45280ff700 2023-07-20T19:03:37.448 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.388+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4518003f90 con 0x7f45280ff700 2023-07-20T19:03:37.448 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.389+0000 7f45257fa700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f45100a11d0 0x7f45100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:37.449 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.389+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f4518014070 con 0x7f45280ff700 2023-07-20T19:03:37.449 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.389+0000 7f45277fe700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f45100a11d0 0x7f45100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:37.449 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.390+0000 7f45277fe700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f45100a11d0 0x7f45100a3690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f451c006010 tx=0x7f451c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:03:37.449 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.392+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4518096480 con 0x7f45280ff700 2023-07-20T19:03:37.467 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:37 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:03:37.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:37 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:03:37.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:37 smithi061 ceph-mon[127454]: osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816] boot 2023-07-20T19:03:37.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:37 smithi061 ceph-mon[127454]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T19:03:37.468 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:37.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:37 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:03:37.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:37 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:03:37.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:37 smithi149 ceph-mon[128885]: osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816] boot 2023-07-20T19:03:37.552 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:37 smithi149 ceph-mon[128885]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T19:03:37.553 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:37.645 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:37.589+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4528060bb0 con 0x7f45100a11d0 2023-07-20T19:03:37.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:37 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:03:37.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:37 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:03:37.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:37 smithi156 ceph-mon[129228]: osd.4 [v2:172.21.15.149:6808/3513915816,v1:172.21.15.149:6809/3513915816] boot 2023-07-20T19:03:37.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:37 smithi156 ceph-mon[129228]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T19:03:37.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T19:03:38.763 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:38 smithi156 ceph-mon[129228]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:38.763 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:38 smithi156 ceph-mon[129228]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T19:03:38.764 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:38.764 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:38.764 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:38.764 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:38 smithi156 ceph-mon[129228]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:38 smithi061 ceph-mon[127454]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:38 smithi061 ceph-mon[127454]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T19:03:38.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:38 smithi061 ceph-mon[127454]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:39.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:38 smithi149 ceph-mon[128885]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:39.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:38 smithi149 ceph-mon[128885]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T19:03:39.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:39.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:39.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:39.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:38 smithi149 ceph-mon[128885]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:40.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:39 smithi149 ceph-mon[128885]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:40.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:39 smithi156 ceph-mon[129228]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:40.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:39 smithi061 ceph-mon[127454]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:41.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:40 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/249355620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]: dispatch 2023-07-20T19:03:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:40 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]: dispatch 2023-07-20T19:03:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:40 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]': finished 2023-07-20T19:03:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:40 smithi149 ceph-mon[128885]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T19:03:41.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:40 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/4150007042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:03:41.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:40 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/249355620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]: dispatch 2023-07-20T19:03:41.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:40 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]: dispatch 2023-07-20T19:03:41.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:40 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]': finished 2023-07-20T19:03:41.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:40 smithi156 ceph-mon[129228]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T19:03:41.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:40 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:41.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:40 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/4150007042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:03:41.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:40 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/249355620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]: dispatch 2023-07-20T19:03:41.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:40 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]: dispatch 2023-07-20T19:03:41.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:40 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fae1b15-43cb-40d9-ad90-73af4cc96197"}]': finished 2023-07-20T19:03:41.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:40 smithi061 ceph-mon[127454]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T19:03:41.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:40 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:41.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:40 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/4150007042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:03:43.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:43 smithi149 ceph-mon[128885]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:43.854 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:43 smithi156 ceph-mon[129228]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:43.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:43 smithi061 ceph-mon[127454]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:45.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:45 smithi149 ceph-mon[128885]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:45.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:45 smithi156 ceph-mon[129228]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:45.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:45 smithi061 ceph-mon[127454]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:46.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:46 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T19:03:46.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:46 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:46.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:46 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T19:03:46.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:46 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:46.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:46 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T19:03:46.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:46 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:47.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:47 smithi156 ceph-mon[129228]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:47.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:47 smithi156 ceph-mon[129228]: Deploying daemon osd.5 on smithi156 2023-07-20T19:03:47.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:47 smithi149 ceph-mon[128885]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:47.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:47 smithi149 ceph-mon[128885]: Deploying daemon osd.5 on smithi156 2023-07-20T19:03:47.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:47 smithi061 ceph-mon[127454]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:47.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:47 smithi061 ceph-mon[127454]: Deploying daemon osd.5 on smithi156 2023-07-20T19:03:49.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:49 smithi149 ceph-mon[128885]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:49.903 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:49 smithi156 ceph-mon[129228]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:49.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:49 smithi061 ceph-mon[127454]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:51.702 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:51 smithi156 ceph-mon[129228]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:51.702 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:51 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:51.703 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:51 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:51.703 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:51 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:51.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:51 smithi061 ceph-mon[127454]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:51.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:51 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:51.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:51 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:51.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:51 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:52.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:51 smithi149 ceph-mon[128885]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:52.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:51 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:52.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:51 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:52.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:51 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:53.208 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.151+0000 7f45257fa700 1 -- 172.21.15.156:0/2481539298 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4528060bb0 con 0x7f45100a11d0 2023-07-20T19:03:53.208 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 5 on host 'smithi156' 2023-07-20T19:03:53.210 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.153+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f45100a11d0 msgr2=0x7f45100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:53.210 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.153+0000 7f452e6ba700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f45100a11d0 0x7f45100a3690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f451c006010 tx=0x7f451c00b260 comp rx=0 tx=0).stop 2023-07-20T19:03:53.210 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.153+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 msgr2=0x7f4528069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:53.210 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.153+0000 7f452e6ba700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 0x7f4528069620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f451800e970 tx=0x7f451800ed30 comp rx=0 tx=0).stop 2023-07-20T19:03:53.212 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 shutdown_connections 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f45100a11d0 0x7f45100a3690 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45281012a0 0x7f452819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45280ff700 0x7f4528069620 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 --2- 172.21.15.156:0/2481539298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4528100900 0x7f452819b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 >> 172.21.15.156:0/2481539298 conn(0x7f45280fac80 msgr2=0x7f4528103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 shutdown_connections 2023-07-20T19:03:53.213 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:03:53.154+0000 7f452e6ba700 1 -- 172.21.15.156:0/2481539298 wait complete. 2023-07-20T19:03:53.676 DEBUG:teuthology.orchestra.run.smithi156:osd.5> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T19:03:53.679 INFO:tasks.cephadm:Deploying osd.6 on smithi061 with /dev/vg_nvme/lv_2... 2023-07-20T19:03:53.680 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T19:03:53.721 INFO:journalctl@ceph.osd.5.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:03:53.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:53.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:53.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:53.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:53.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:53.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:54.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:54.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:54.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:54.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:54.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:54.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:54.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:54.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:54.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:03:54.270 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:03:54.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:54 smithi061 ceph-mon[127454]: from='osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T19:03:54.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:54 smithi061 ceph-mon[127454]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T19:03:54.909 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:54 smithi156 ceph-mon[129228]: from='osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T19:03:54.909 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:54 smithi156 ceph-mon[129228]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T19:03:54.909 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:03:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:03:54.513+0000 7f24ef9bd640 -1 osd.5 0 log_to_monitors true 2023-07-20T19:03:55.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:54 smithi149 ceph-mon[128885]: from='osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T19:03:55.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:54 smithi149 ceph-mon[128885]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T19:03:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T19:03:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T19:03:55.762 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 10+0 records in 2023-07-20T19:03:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:10+0 records out 2023-07-20T19:03:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112788 s, 930 MB/s 2023-07-20T19:03:55.763 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 2023-07-20T19:03:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping successful for: 2023-07-20T19:03:55.830 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:03:55.568+0000 7f24e5035700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T19:03:55.830 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:03:55.574+0000 7f24dce23700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:03:55.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-mon[129228]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:55.831 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-mon[129228]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T19:03:55.831 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-mon[129228]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T19:03:55.831 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:55.831 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-mon[129228]: from='osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:03:55.831 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:55 smithi156 ceph-mon[129228]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:03:55.915 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:55 smithi061 ceph-mon[127454]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:55.916 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:55 smithi061 ceph-mon[127454]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T19:03:55.916 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:55 smithi061 ceph-mon[127454]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T19:03:55.916 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:55 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:55.916 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:55 smithi061 ceph-mon[127454]: from='osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:03:55.916 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:55 smithi061 ceph-mon[127454]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:03:56.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:55 smithi149 ceph-mon[128885]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:56.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:55 smithi149 ceph-mon[128885]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T19:03:56.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:55 smithi149 ceph-mon[128885]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T19:03:56.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:55 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:56.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:55 smithi149 ceph-mon[128885]: from='osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:03:56.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:55 smithi149 ceph-mon[128885]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:03:56.129 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi061:vg_nvme/lv_2 2023-07-20T19:03:56.687 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:03:56.896 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:03:56.897 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T19:03:56.897 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:56.897 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:56.897 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.897 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.898 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:56.898 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:56.898 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.898 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:56.898 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:56.898 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:56 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:03:56.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T19:03:56.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:56.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:56.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:56.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:56 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:57.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:03:57.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T19:03:57.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:57.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:57.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:03:57.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:56 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:03:57.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:03:57.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:03:57.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:57.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi156 2023-07-20T19:03:57.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi156 to 8584M 2023-07-20T19:03:57.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:57.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844] boot 2023-07-20T19:03:57.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T19:03:57.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:57.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:57 smithi061 ceph-mon[127454]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T19:03:57.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:03:57.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:03:57.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:57.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi156 2023-07-20T19:03:57.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi156 to 8584M 2023-07-20T19:03:57.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:57.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844] boot 2023-07-20T19:03:57.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T19:03:57.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:57.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:57 smithi156 ceph-mon[129228]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T19:03:58.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:03:58.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:03:58.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T19:03:58.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi156 2023-07-20T19:03:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi156 to 8584M 2023-07-20T19:03:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: osd.5 [v2:172.21.15.156:6808/211953844,v1:172.21.15.156:6809/211953844] boot 2023-07-20T19:03:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T19:03:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T19:03:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:57 smithi149 ceph-mon[128885]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T19:03:58.110 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 -- 172.21.15.61:0/3013872304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c100810 msgr2=0x7ffa5c100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:58.110 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/3013872304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c100810 0x7ffa5c100c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffa500099c0 tx=0x7ffa50009cd0 comp rx=0 tx=0).stop 2023-07-20T19:03:58.110 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 -- 172.21.15.61:0/3013872304 shutdown_connections 2023-07-20T19:03:58.110 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/3013872304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:58.111 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/3013872304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c100810 0x7ffa5c100c30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:58.111 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/3013872304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 0x7ffa5c101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:58.111 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 -- 172.21.15.61:0/3013872304 >> 172.21.15.61:0/3013872304 conn(0x7ffa5c0fbd90 msgr2=0x7ffa5c0fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:03:58.111 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 -- 172.21.15.61:0/3013872304 shutdown_connections 2023-07-20T19:03:58.111 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.054+0000 7ffa61a5a700 1 -- 172.21.15.61:0/3013872304 wait complete. 2023-07-20T19:03:58.113 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 Processor -- start 2023-07-20T19:03:58.113 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 -- start start 2023-07-20T19:03:58.113 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa5c100810 0x7ffa5c074510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:58.113 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 0x7ffa5c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:58.114 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:58.114 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c073820 con 0x7ffa5c1023b0 2023-07-20T19:03:58.114 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c073990 con 0x7ffa5c100810 2023-07-20T19:03:58.114 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa61a5a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c1a05f0 con 0x7ffa5c101a10 2023-07-20T19:03:58.115 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:58.115 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c0730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57788/0 (socket says 172.21.15.61:57788) 2023-07-20T19:03:58.115 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 -- 172.21.15.61:0/1685721404 learned_addr learned my addr 172.21.15.61:0/1685721404 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:03:58.115 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5a7fc700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 0x7ffa5c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:58.115 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5affd700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa5c100810 0x7ffa5c074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:58.116 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 -- 172.21.15.61:0/1685721404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 msgr2=0x7ffa5c072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:58.116 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 0x7ffa5c072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:58.116 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 -- 172.21.15.61:0/1685721404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa5c100810 msgr2=0x7ffa5c074510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:03:58.116 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa5c100810 0x7ffa5c074510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:03:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5b7fe700 1 -- 172.21.15.61:0/1685721404 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500095f0 con 0x7ffa5c1023b0 2023-07-20T19:03:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.055+0000 7ffa5a7fc700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 0x7ffa5c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:03:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.056+0000 7ffa5b7fe700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c0730a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400ada0 tx=0x7ffa5400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:03:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.056+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5400f070 con 0x7ffa5c1023b0 2023-07-20T19:03:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.056+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5c1a08d0 con 0x7ffa5c1023b0 2023-07-20T19:03:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.056+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa54007b30 con 0x7ffa5c1023b0 2023-07-20T19:03:58.118 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.056+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa54005440 con 0x7ffa5c1023b0 2023-07-20T19:03:58.118 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.056+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5c1a0dd0 con 0x7ffa5c1023b0 2023-07-20T19:03:58.118 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.057+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5c10c090 con 0x7ffa5c1023b0 2023-07-20T19:03:58.119 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.058+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffa54004ad0 con 0x7ffa5c1023b0 2023-07-20T19:03:58.119 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.058+0000 7ffa60a58700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa440a11d0 0x7ffa440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:03:58.120 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.059+0000 7ffa5affd700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa440a11d0 0x7ffa440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:03:58.120 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.059+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7ffa540cb190 con 0x7ffa5c1023b0 2023-07-20T19:03:58.120 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.059+0000 7ffa5affd700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa440a11d0 0x7ffa440a3690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffa5000b580 tx=0x7ffa50005320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:03:58.120 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.063+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffa54095770 con 0x7ffa5c1023b0 2023-07-20T19:03:58.325 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:03:58.269+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa5c1a1410 con 0x7ffa440a11d0 2023-07-20T19:03:58.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:58 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:58.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:58 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:58.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:58 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:58.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:58 smithi156 ceph-mon[129228]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T19:03:58.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:58 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:58.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:58 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:58.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:58 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:58.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:58 smithi061 ceph-mon[127454]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T19:03:59.051 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:58 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:03:59.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:58 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:03:59.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:58 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:03:59.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:58 smithi149 ceph-mon[128885]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T19:03:59.632 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:59 smithi061 ceph-mon[127454]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:03:59.632 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:59 smithi061 ceph-mon[127454]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:59.633 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:03:59 smithi061 ceph-mon[127454]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T19:03:59.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:59 smithi156 ceph-mon[129228]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:03:59.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:59 smithi156 ceph-mon[129228]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:03:59.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:03:59 smithi156 ceph-mon[129228]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T19:04:00.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:59 smithi149 ceph-mon[128885]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:00.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:59 smithi149 ceph-mon[128885]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:00.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:03:59 smithi149 ceph-mon[128885]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T19:04:01.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:01.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T19:04:01.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3765841487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84316d15-3a51-432c-9d02-57270ac09769"}]: dispatch 2023-07-20T19:04:01.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3765841487' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84316d15-3a51-432c-9d02-57270ac09769"}]': finished 2023-07-20T19:04:01.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T19:04:01.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:01.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:01 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/912618119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:01.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:01.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T19:04:01.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3765841487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84316d15-3a51-432c-9d02-57270ac09769"}]: dispatch 2023-07-20T19:04:01.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3765841487' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84316d15-3a51-432c-9d02-57270ac09769"}]': finished 2023-07-20T19:04:01.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T19:04:01.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:01.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:01 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/912618119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T19:04:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3765841487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84316d15-3a51-432c-9d02-57270ac09769"}]: dispatch 2023-07-20T19:04:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3765841487' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84316d15-3a51-432c-9d02-57270ac09769"}]': finished 2023-07-20T19:04:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T19:04:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:01 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/912618119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:03.801 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:03 smithi149 ceph-mon[128885]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:03.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:03 smithi156 ceph-mon[129228]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:03.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:03 smithi061 ceph-mon[127454]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:05.797 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:05 smithi061 ceph-mon[127454]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2023-07-20T19:04:05.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:05 smithi156 ceph-mon[129228]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2023-07-20T19:04:06.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:05 smithi149 ceph-mon[128885]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2023-07-20T19:04:07.905 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:07 smithi061 ceph-mon[127454]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-07-20T19:04:07.906 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T19:04:07.906 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:07 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:07.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:07 smithi156 ceph-mon[129228]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-07-20T19:04:07.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T19:04:07.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:07 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:08.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:07 smithi149 ceph-mon[128885]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-07-20T19:04:08.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T19:04:08.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:07 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:08.717 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:08 smithi061 ceph-mon[127454]: Deploying daemon osd.6 on smithi061 2023-07-20T19:04:08.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:08 smithi156 ceph-mon[129228]: Deploying daemon osd.6 on smithi061 2023-07-20T19:04:09.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:08 smithi149 ceph-mon[128885]: Deploying daemon osd.6 on smithi061 2023-07-20T19:04:09.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:09 smithi061 ceph-mon[127454]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T19:04:09.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:09 smithi156 ceph-mon[129228]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T19:04:10.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:09 smithi149 ceph-mon[128885]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T19:04:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:11 smithi156 ceph-mon[129228]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T19:04:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:11.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:11 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:11.967 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:11 smithi061 ceph-mon[127454]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T19:04:11.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:11.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:11.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:11 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:12.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:11 smithi149 ceph-mon[128885]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T19:04:12.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:12.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:12.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:11 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:13.891 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:13 smithi061 ceph-mon[127454]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T19:04:13.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:13 smithi156 ceph-mon[129228]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T19:04:13.924 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.868+0000 7ffa60a58700 1 -- 172.21.15.61:0/1685721404 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffa5c1a1410 con 0x7ffa440a11d0 2023-07-20T19:04:13.925 INFO:teuthology.orchestra.run.smithi061.stdout:Created osd(s) 6 on host 'smithi061' 2023-07-20T19:04:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa440a11d0 msgr2=0x7ffa440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa440a11d0 0x7ffa440a3690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffa5000b580 tx=0x7ffa50005320 comp rx=0 tx=0).stop 2023-07-20T19:04:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 msgr2=0x7ffa5c0730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c0730a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400ada0 tx=0x7ffa5400c330 comp rx=0 tx=0).stop 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 shutdown_connections 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa440a11d0 0x7ffa440a3690 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa5c1023b0 0x7ffa5c0730a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa5c100810 0x7ffa5c074510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 --2- 172.21.15.61:0/1685721404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c101a10 0x7ffa5c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 >> 172.21.15.61:0/1685721404 conn(0x7ffa5c0fbd90 msgr2=0x7ffa5c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 shutdown_connections 2023-07-20T19:04:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:04:13.872+0000 7ffa61a5a700 1 -- 172.21.15.61:0/1685721404 wait complete. 2023-07-20T19:04:14.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:13 smithi149 ceph-mon[128885]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-07-20T19:04:14.399 DEBUG:teuthology.orchestra.run.smithi061:osd.6> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T19:04:14.402 INFO:tasks.cephadm:Deploying osd.7 on smithi149 with /dev/vg_nvme/lv_2... 2023-07-20T19:04:14.402 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T19:04:14.438 INFO:journalctl@ceph.osd.6.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:04:14.977 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:04:15.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:14 smithi149 ceph-mon[128885]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T19:04:15.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:14 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.164 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:14 smithi061 ceph-mon[127454]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T19:04:15.164 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.165 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.165 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.165 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:14 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:14 smithi156 ceph-mon[129228]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T19:04:15.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:14 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:15.467 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:04:15 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:04:15.107+0000 7f6931fcb640 -1 osd.6 0 log_to_monitors true 2023-07-20T19:04:16.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:15 smithi149 ceph-mon[128885]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T19:04:16.129 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:15 smithi061 ceph-mon[127454]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T19:04:16.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:15 smithi156 ceph-mon[129228]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T19:04:16.796 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T19:04:16.796 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T19:04:16.797 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2023-07-20T19:04:16.797 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2023-07-20T19:04:16.797 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111017 s, 945 MB/s 2023-07-20T19:04:16.797 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2023-07-20T19:04:16.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-mon[127454]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:16.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-mon[127454]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T19:04:16.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-mon[127454]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T19:04:16.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:16.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-mon[127454]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:04:16.969 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:04:16.710+0000 7f6927643700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T19:04:16.970 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:04:16 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:04:16.723+0000 7f691fc32700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:04:17.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:16 smithi149 ceph-mon[128885]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:17.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:16 smithi149 ceph-mon[128885]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T19:04:17.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:16 smithi149 ceph-mon[128885]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T19:04:17.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:16 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:17.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:16 smithi149 ceph-mon[128885]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:04:17.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:16 smithi156 ceph-mon[129228]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:17.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:16 smithi156 ceph-mon[129228]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T19:04:17.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:16 smithi156 ceph-mon[129228]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T19:04:17.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:16 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:17.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:16 smithi156 ceph-mon[129228]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:04:17.237 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi149:vg_nvme/lv_2 2023-07-20T19:04:17.792 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:04:18.031 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:04:18.032 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T19:04:18.032 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:18.032 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:18.033 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi061 2023-07-20T19:04:18.033 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.033 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.033 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.033 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.034 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.034 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi061 to 5722M 2023-07-20T19:04:18.034 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.034 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:18.034 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:18.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:18.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:18.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:18.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:04:18.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T19:04:18.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:18.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:18.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi061 2023-07-20T19:04:18.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi061 to 5722M 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:18.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:18.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:18.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:18.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:04:18.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T19:04:18.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:18.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:18.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi061 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi061 to 5722M 2023-07-20T19:04:18.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:18.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:19.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:04:19.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:04:19.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:19.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:19.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297] boot 2023-07-20T19:04:19.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T19:04:19.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:18 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:19.069 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 -- 172.21.15.149:0/4145064155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1148100750 msgr2=0x7f1148100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:19.070 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 --2- 172.21.15.149:0/4145064155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1148100750 0x7f1148100bb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11440099c0 tx=0x7f1144009cd0 comp rx=0 tx=0).stop 2023-07-20T19:04:19.070 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 -- 172.21.15.149:0/4145064155 shutdown_connections 2023-07-20T19:04:19.070 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 --2- 172.21.15.149:0/4145064155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11481010f0 0x7f11481079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:19.070 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 --2- 172.21.15.149:0/4145064155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11480ff550 0x7f11480ff970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:19.070 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 --2- 172.21.15.149:0/4145064155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1148100750 0x7f1148100bb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:19.071 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 -- 172.21.15.149:0/4145064155 >> 172.21.15.149:0/4145064155 conn(0x7f11480fab10 msgr2=0x7f11480fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:04:19.071 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 -- 172.21.15.149:0/4145064155 shutdown_connections 2023-07-20T19:04:19.071 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.012+0000 7f114f1a2700 1 -- 172.21.15.149:0/4145064155 wait complete. 2023-07-20T19:04:19.072 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 Processor -- start 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 -- start start 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11480ff550 0x7f1148068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1148100750 0x7f114819b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 0x7f114819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114819dea0 con 0x7f11480ff550 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114819e010 con 0x7f11481010f0 2023-07-20T19:04:19.073 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114f1a2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114819e1d0 con 0x7f1148100750 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114e9a1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 0x7f114819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114e9a1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 0x7f114819d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:44238/0 (socket says 172.21.15.149:44238) 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114e9a1700 1 -- 172.21.15.149:0/575897133 learned_addr learned my addr 172.21.15.149:0/575897133 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114d99f700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1148100750 0x7f114819b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.013+0000 7f114e1a0700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11480ff550 0x7f1148068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114e9a1700 1 -- 172.21.15.149:0/575897133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1148100750 msgr2=0x7f114819b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:19.074 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114e9a1700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1148100750 0x7f114819b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114e9a1700 1 -- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11480ff550 msgr2=0x7f1148068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114e9a1700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11480ff550 0x7f1148068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114e9a1700 1 -- 172.21.15.149:0/575897133 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11440095f0 con 0x7f11481010f0 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114e9a1700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 0x7f114819d700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f114000b970 tx=0x7f114000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mon.2 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114000d860 con 0x7f11481010f0 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mon.2 v2:172.21.15.149:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f114000de80 con 0x7f11481010f0 2023-07-20T19:04:19.075 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mon.2 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140017400 con 0x7f11481010f0 2023-07-20T19:04:19.076 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114819e4b0 con 0x7f11481010f0 2023-07-20T19:04:19.076 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.014+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114819e930 con 0x7f11481010f0 2023-07-20T19:04:19.076 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.015+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114810add0 con 0x7f11481010f0 2023-07-20T19:04:19.076 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.016+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mon.2 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f114000e630 con 0x7f11481010f0 2023-07-20T19:04:19.077 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.017+0000 7f113f7fe700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f11340a11d0 0x7f11340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:19.077 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.017+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mon.2 v2:172.21.15.149:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f1140006e40 con 0x7f11481010f0 2023-07-20T19:04:19.077 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.017+0000 7f114e1a0700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f11340a11d0 0x7f11340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:19.077 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.018+0000 7f114e1a0700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f11340a11d0 0x7f11340a3690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f11481005b0 tx=0x7f113800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:04:19.078 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.021+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mon.2 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1140095e20 con 0x7f11481010f0 2023-07-20T19:04:19.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:04:19.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:04:19.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:19.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:19.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297] boot 2023-07-20T19:04:19.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T19:04:19.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:18 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:19.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:04:19.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:04:19.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T19:04:19.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:19.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: osd.6 [v2:172.21.15.61:6818/1770665297,v1:172.21.15.61:6819/1770665297] boot 2023-07-20T19:04:19.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T19:04:19.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:18 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T19:04:19.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:19.214+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1148060bb0 con 0x7f11340a11d0 2023-07-20T19:04:20.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:19 smithi149 ceph-mon[128885]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T19:04:20.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:19 smithi149 ceph-mon[128885]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:20.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:04:20.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:04:20.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:20.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:19 smithi156 ceph-mon[129228]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T19:04:20.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:19 smithi156 ceph-mon[129228]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:20.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:04:20.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:04:20.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:20.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:19 smithi061 ceph-mon[127454]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T19:04:20.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:19 smithi061 ceph-mon[127454]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:20.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:04:20.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:04:20.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:21.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:20 smithi149 ceph-mon[128885]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:21.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:20 smithi156 ceph-mon[129228]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:21.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:20 smithi061 ceph-mon[127454]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:22.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:21 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/3647608717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]: dispatch 2023-07-20T19:04:22.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:21 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]: dispatch 2023-07-20T19:04:22.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:21 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]': finished 2023-07-20T19:04:22.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:21 smithi149 ceph-mon[128885]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T19:04:22.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:22.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:21 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/3647608717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]: dispatch 2023-07-20T19:04:22.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:21 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]: dispatch 2023-07-20T19:04:22.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:21 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]': finished 2023-07-20T19:04:22.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:21 smithi156 ceph-mon[129228]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T19:04:22.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:22.217 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:21 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/3647608717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]: dispatch 2023-07-20T19:04:22.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:21 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]: dispatch 2023-07-20T19:04:22.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:21 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "705c1527-9342-402c-add4-4382e005dbfe"}]': finished 2023-07-20T19:04:22.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:21 smithi061 ceph-mon[127454]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T19:04:22.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:23.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:22 smithi149 ceph-mon[128885]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:23.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:22 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/2321800356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:23.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:22 smithi156 ceph-mon[129228]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:23.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:22 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/2321800356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:23.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:22 smithi061 ceph-mon[127454]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:23.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:22 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/2321800356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:25.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:24 smithi149 ceph-mon[128885]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:25.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:24 smithi156 ceph-mon[129228]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:25.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:24 smithi061 ceph-mon[127454]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:26.955 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:26 smithi149 ceph-mon[128885]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:27.165 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:26 smithi156 ceph-mon[129228]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:27.218 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:26 smithi061 ceph-mon[127454]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:28.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T19:04:28.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:28.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T19:04:28.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:28.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T19:04:28.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:29.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:29 smithi149 ceph-mon[128885]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:29.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:29 smithi149 ceph-mon[128885]: Deploying daemon osd.7 on smithi149 2023-07-20T19:04:29.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:29 smithi156 ceph-mon[129228]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:29.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:29 smithi156 ceph-mon[129228]: Deploying daemon osd.7 on smithi149 2023-07-20T19:04:29.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:29 smithi061 ceph-mon[127454]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:29.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:29 smithi061 ceph-mon[127454]: Deploying daemon osd.7 on smithi149 2023-07-20T19:04:31.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:31 smithi156 ceph-mon[129228]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:31.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:31 smithi149 ceph-mon[128885]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:31.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:31 smithi061 ceph-mon[127454]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:33.905 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:33 smithi149 ceph-mon[128885]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:33.905 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:33.906 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:33.906 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:33 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:33.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:33 smithi156 ceph-mon[129228]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:33.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:33.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:33.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:33 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:33.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:33 smithi061 ceph-mon[127454]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:33.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:33.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:33.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:33 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.591 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.535+0000 7f113f7fe700 1 -- 172.21.15.149:0/575897133 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1148060bb0 con 0x7f11340a11d0 2023-07-20T19:04:35.592 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 7 on host 'smithi149' 2023-07-20T19:04:35.594 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f11340a11d0 msgr2=0x7f11340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:35.594 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f11340a11d0 0x7f11340a3690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f11481005b0 tx=0x7f113800b410 comp rx=0 tx=0).stop 2023-07-20T19:04:35.595 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 msgr2=0x7f114819d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:35.595 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 0x7f114819d700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f114000b970 tx=0x7f114000bc80 comp rx=0 tx=0).stop 2023-07-20T19:04:35.595 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 shutdown_connections 2023-07-20T19:04:35.596 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f11340a11d0 0x7f11340a3690 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:35.596 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11481010f0 0x7f114819d700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:35.597 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11480ff550 0x7f1148068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:35.597 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 --2- 172.21.15.149:0/575897133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1148100750 0x7f114819b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:35.597 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 >> 172.21.15.149:0/575897133 conn(0x7f11480fab10 msgr2=0x7f11480fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:04:35.597 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 shutdown_connections 2023-07-20T19:04:35.597 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:04:35.537+0000 7f114f1a2700 1 -- 172.21.15.149:0/575897133 wait complete. 2023-07-20T19:04:35.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:35 smithi149 ceph-mon[128885]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:35.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:35 smithi156 ceph-mon[129228]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:35 smithi061 ceph-mon[127454]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:35.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:35.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:36.015 DEBUG:teuthology.orchestra.run.smithi149:osd.7> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T19:04:36.017 INFO:tasks.cephadm:Deploying osd.8 on smithi156 with /dev/vg_nvme/lv_2... 2023-07-20T19:04:36.017 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T19:04:36.047 INFO:journalctl@ceph.osd.7.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:04:36.581 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:04:36.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:04:36 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:04:36.613+0000 7f685f864640 -1 osd.7 0 log_to_monitors true 2023-07-20T19:04:37.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:37 smithi149 ceph-mon[128885]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:37.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:37 smithi149 ceph-mon[128885]: from='osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T19:04:37.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:37 smithi149 ceph-mon[128885]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T19:04:37.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:37 smithi156 ceph-mon[129228]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:37.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:37 smithi156 ceph-mon[129228]: from='osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T19:04:37.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:37 smithi156 ceph-mon[129228]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T19:04:37.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:37 smithi061 ceph-mon[127454]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:37.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:37 smithi061 ceph-mon[127454]: from='osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T19:04:37.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:37 smithi061 ceph-mon[127454]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T19:04:38.150 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T19:04:38.151 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T19:04:38.151 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-20T19:04:38.151 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-20T19:04:38.151 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00822649 s, 1.3 GB/s 2023-07-20T19:04:38.151 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-20T19:04:38.538 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2023-07-20T19:04:38.802 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:04:38.520+0000 7f6854edc700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T19:04:38.803 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:04:38.528+0000 7f684ccca700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:04:38.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T19:04:38.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T19:04:38.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:38.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:04:38.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:38.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:38.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T19:04:38.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T19:04:38.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:38.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:04:38.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:04:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:38.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:38.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T19:04:38.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T19:04:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:04:38.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:04:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:38.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:38.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:38.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:38.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:39.099 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:04:39.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:39.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi149 2023-07-20T19:04:39.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi149 to 1959M 2023-07-20T19:04:39.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:04:39.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T19:04:39.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:39.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:39.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:39.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:39.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:39.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:39.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:39.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi149 2023-07-20T19:04:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi149 to 1959M 2023-07-20T19:04:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:04:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T19:04:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:40.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T19:04:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi149 2023-07-20T19:04:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi149 to 1959M 2023-07-20T19:04:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:04:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T19:04:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:40.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:40.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:40.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:40.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:40.317 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/299044648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc0101a30 msgr2=0x7f8cc0101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:40.317 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/299044648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc0101a30 0x7f8cc0101e90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009990 tx=0x7f8cbc009ca0 comp rx=0 tx=0).stop 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/299044648 shutdown_connections 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/299044648 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8cc01023d0 0x7f8cc0108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/299044648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cc0100830 0x7f8cc0100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/299044648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc0101a30 0x7f8cc0101e90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/299044648 >> 172.21.15.156:0/299044648 conn(0x7f8cc00fbdf0 msgr2=0x7f8cc00fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/299044648 shutdown_connections 2023-07-20T19:04:40.318 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/299044648 wait complete. 2023-07-20T19:04:40.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.260+0000 7f8cc7b0e700 1 Processor -- start 2023-07-20T19:04:40.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 -- start start 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cc0100830 0x7f8cc01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8cc0101a30 0x7f8cc0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 0x7f8cc019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc01a0260 con 0x7f8cc0100830 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc01a03a0 con 0x7f8cc0101a30 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc01a0530 con 0x7f8cc01023d0 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 0x7f8cc019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:40.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 0x7f8cc019fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48578/0 (socket says 172.21.15.156:48578) 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 -- 172.21.15.156:0/2344042093 learned_addr learned my addr 172.21.15.156:0/2344042093 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc58aa700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cc0100830 0x7f8cc01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc50a9700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8cc0101a30 0x7f8cc0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 -- 172.21.15.156:0/2344042093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8cc0101a30 msgr2=0x7f8cc0198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8cc0101a30 0x7f8cc0198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 -- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cc0100830 msgr2=0x7f8cc01986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:40.322 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cc0100830 0x7f8cc01986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 -- 172.21.15.156:0/2344042093 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0095f0 con 0x7f8cc01023d0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc60ab700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 0x7f8cc019fc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800ef60 tx=0x7f8cb800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8018070 con 0x7f8cc01023d0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc01a07b0 con 0x7f8cc01023d0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.261+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cb800f040 con 0x7f8cc01023d0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.262+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb800cd70 con 0x7f8cc01023d0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.262+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc01a0c60 con 0x7f8cc01023d0 2023-07-20T19:04:40.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.263+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc010c030 con 0x7f8cc01023d0 2023-07-20T19:04:40.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.264+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8cb8003a00 con 0x7f8cc01023d0 2023-07-20T19:04:40.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.264+0000 7f8cb6ffd700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8cac0a1150 0x7f8cac0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:04:40.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.264+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5430+0+0 (secure 0 0 0) 0x7f8cb80cc340 con 0x7f8cc01023d0 2023-07-20T19:04:40.326 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.264+0000 7f8cc58aa700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8cac0a1150 0x7f8cac0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:04:40.326 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.265+0000 7f8cc58aa700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8cac0a1150 0x7f8cac0a3610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0101890 tx=0x7f8cb000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:04:40.327 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.268+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8cb8096660 con 0x7f8cc01023d0 2023-07-20T19:04:40.516 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:40.459+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc0108390 con 0x7f8cac0a1150 2023-07-20T19:04:40.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:04:40.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:04:40.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817] boot 2023-07-20T19:04:40.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T19:04:40.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:40.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:04:40.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:04:40.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:40 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:40.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:04:40.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:04:40.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817] boot 2023-07-20T19:04:40.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T19:04:40.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:40.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:04:40.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:04:40.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:40 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:04:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:04:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: osd.7 [v2:172.21.15.149:6816/1095358817,v1:172.21.15.149:6817/1095358817] boot 2023-07-20T19:04:41.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T19:04:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T19:04:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:04:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:04:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:40 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:41.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:41 smithi156 ceph-mon[129228]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:41.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:41 smithi156 ceph-mon[129228]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:41.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:41 smithi156 ceph-mon[129228]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T19:04:41.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:41 smithi156 ceph-mon[129228]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T19:04:41.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:41 smithi061 ceph-mon[127454]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:41.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:41 smithi061 ceph-mon[127454]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:41.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:41 smithi061 ceph-mon[127454]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T19:04:41.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:41 smithi061 ceph-mon[127454]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T19:04:42.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:41 smithi149 ceph-mon[128885]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:42.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:41 smithi149 ceph-mon[128885]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:04:42.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:41 smithi149 ceph-mon[128885]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T19:04:42.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:41 smithi149 ceph-mon[128885]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T19:04:43.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:43.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T19:04:43.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/508085596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]: dispatch 2023-07-20T19:04:43.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]: dispatch 2023-07-20T19:04:43.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]': finished 2023-07-20T19:04:43.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T19:04:43.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:04:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T19:04:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/508085596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]: dispatch 2023-07-20T19:04:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]: dispatch 2023-07-20T19:04:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]': finished 2023-07-20T19:04:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T19:04:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:04:43.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T19:04:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/508085596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]: dispatch 2023-07-20T19:04:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]: dispatch 2023-07-20T19:04:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e790866-ee13-4c27-bbd4-ea90e853de1f"}]': finished 2023-07-20T19:04:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T19:04:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:04:44.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:44 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/3553676274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:44.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:44 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/3553676274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:45.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:44 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/3553676274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:04:45.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:45 smithi156 ceph-mon[129228]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:45.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:45 smithi061 ceph-mon[127454]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:46.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:45 smithi149 ceph-mon[128885]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:47.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:47 smithi156 ceph-mon[129228]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:47.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:47 smithi061 ceph-mon[127454]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:48.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:47 smithi149 ceph-mon[128885]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:49.915 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:49 smithi156 ceph-mon[129228]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:49.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:49 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T19:04:49.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:49 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:49.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:49 smithi061 ceph-mon[127454]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:49.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:49 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T19:04:49.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:49 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:50.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:49 smithi149 ceph-mon[128885]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:50.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:49 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T19:04:50.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:49 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:50.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:50 smithi156 ceph-mon[129228]: Deploying daemon osd.8 on smithi156 2023-07-20T19:04:50.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:50 smithi061 ceph-mon[127454]: Deploying daemon osd.8 on smithi156 2023-07-20T19:04:51.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:50 smithi149 ceph-mon[128885]: Deploying daemon osd.8 on smithi156 2023-07-20T19:04:51.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:51 smithi156 ceph-mon[129228]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:51.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:51 smithi061 ceph-mon[127454]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:52.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:51 smithi149 ceph-mon[128885]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:53.829 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:53 smithi156 ceph-mon[129228]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:53.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:53 smithi061 ceph-mon[127454]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:54.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:53 smithi149 ceph-mon[128885]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:54.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:54.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:54.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:54 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:54.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:54.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:54.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:54 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:55.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:55.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:55.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:54 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:55.877 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:55 smithi156 ceph-mon[129228]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:55.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:55 smithi061 ceph-mon[127454]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:56.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:55 smithi149 ceph-mon[128885]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:56.730 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 8 on host 'smithi156' 2023-07-20T19:04:56.731 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.673+0000 7f8cb6ffd700 1 -- 172.21.15.156:0/2344042093 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8cc0108390 con 0x7f8cac0a1150 2023-07-20T19:04:56.733 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.675+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8cac0a1150 msgr2=0x7f8cac0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:56.733 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.675+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8cac0a1150 0x7f8cac0a3610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0101890 tx=0x7f8cb000b410 comp rx=0 tx=0).stop 2023-07-20T19:04:56.733 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.675+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 msgr2=0x7f8cc019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:04:56.733 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.675+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 0x7f8cc019fc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800ef60 tx=0x7f8cb800ef90 comp rx=0 tx=0).stop 2023-07-20T19:04:56.733 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 shutdown_connections 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8cac0a1150 0x7f8cac0a3610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8cc01023d0 0x7f8cc019fc90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cc0100830 0x7f8cc01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 --2- 172.21.15.156:0/2344042093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8cc0101a30 0x7f8cc0198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 >> 172.21.15.156:0/2344042093 conn(0x7f8cc00fbdf0 msgr2=0x7f8cc00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 shutdown_connections 2023-07-20T19:04:56.734 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:04:56.676+0000 7f8cc7b0e700 1 -- 172.21.15.156:0/2344042093 wait complete. 2023-07-20T19:04:57.200 DEBUG:teuthology.orchestra.run.smithi156:osd.8> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T19:04:57.203 INFO:tasks.cephadm:Deploying osd.9 on smithi061 with /dev/vg_nvme/lv_1... 2023-07-20T19:04:57.204 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T19:04:57.232 INFO:journalctl@ceph.osd.8.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:04:57.416 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:57.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:57.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:57.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:57.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:57.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:57.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:57.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:57.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.784 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:04:57.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:57.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:57.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:57.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:04:57.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:57.901 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:04:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:04:57.727+0000 7f4fbc54d640 -1 osd.8 0 log_to_monitors true 2023-07-20T19:04:58.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:58 smithi156 ceph-mon[129228]: from='osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T19:04:58.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:58 smithi156 ceph-mon[129228]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T19:04:58.802 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:58 smithi149 ceph-mon[128885]: from='osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T19:04:58.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:58 smithi149 ceph-mon[128885]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T19:04:58.806 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:58 smithi061 ceph-mon[127454]: from='osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T19:04:58.807 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:58 smithi061 ceph-mon[127454]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T19:04:59.267 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T19:04:59.267 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T19:04:59.267 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 10+0 records in 2023-07-20T19:04:59.268 INFO:teuthology.orchestra.run.smithi061.stderr:10+0 records out 2023-07-20T19:04:59.268 INFO:teuthology.orchestra.run.smithi061.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112331 s, 933 MB/s 2023-07-20T19:04:59.268 INFO:teuthology.orchestra.run.smithi061.stderr:--> Zapping successful for: 2023-07-20T19:04:59.646 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi061:vg_nvme/lv_1 2023-07-20T19:04:59.916 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:04:59.509+0000 7f4fb1bc5700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T19:04:59.916 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:04:59.520+0000 7f4fac1b8700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:04:59.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:59.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:59.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:59.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:59.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:59.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:04:59 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:04:59.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T19:04:59.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T19:04:59.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:04:59.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:04:59.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:04:59.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:59.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:59.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:04:59.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:04:59.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:04:59.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:04:59.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:04:59 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:00.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:05:00.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T19:05:00.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T19:05:00.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:00.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:00.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:00.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:00.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:00.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:04:59 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:00.206 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:05:00.708 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:00 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi156 2023-07-20T19:05:00.708 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:00 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi156 to 5722M 2023-07-20T19:05:00.708 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:00 smithi061 ceph-mon[127454]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:05:00.709 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:00 smithi061 ceph-mon[127454]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T19:05:00.709 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:00.709 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:00.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:00 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi156 2023-07-20T19:05:00.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:00 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi156 to 5722M 2023-07-20T19:05:00.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:00 smithi156 ceph-mon[129228]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:05:00.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:00 smithi156 ceph-mon[129228]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T19:05:00.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:00.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:01.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:00 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi156 2023-07-20T19:05:01.065 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:00 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi156 to 5722M 2023-07-20T19:05:01.065 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:00 smithi149 ceph-mon[128885]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:05:01.065 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:00 smithi149 ceph-mon[128885]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T19:05:01.065 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:01.066 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:01.404 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 -- 172.21.15.61:0/1123143672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4101a00 msgr2=0x7f04b4101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:01.404 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 --2- 172.21.15.61:0/1123143672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4101a00 0x7f04b4101e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f049c0099c0 tx=0x7f049c009cd0 comp rx=0 tx=0).stop 2023-07-20T19:05:01.404 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 -- 172.21.15.61:0/1123143672 shutdown_connections 2023-07-20T19:05:01.404 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 --2- 172.21.15.61:0/1123143672 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 0x7f04b4108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 --2- 172.21.15.61:0/1123143672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4100800 0x7f04b4100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 --2- 172.21.15.61:0/1123143672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4101a00 0x7f04b4101e60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 -- 172.21.15.61:0/1123143672 >> 172.21.15.61:0/1123143672 conn(0x7f04b40fbd60 msgr2=0x7f04b40fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:05:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 -- 172.21.15.61:0/1123143672 shutdown_connections 2023-07-20T19:05:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.346+0000 7f04b99af700 1 -- 172.21.15.61:0/1123143672 wait complete. 2023-07-20T19:05:01.407 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.347+0000 7f04b99af700 1 Processor -- start 2023-07-20T19:05:01.407 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 -- start start 2023-07-20T19:05:01.407 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 0x7f04b41964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:01.407 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4101a00 0x7f04b4196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:01.407 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 0x7f04b419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b419e050 con 0x7f04b4100800 2023-07-20T19:05:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b419e190 con 0x7f04b41023a0 2023-07-20T19:05:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b99af700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b419e320 con 0x7f04b4101a00 2023-07-20T19:05:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 0x7f04b41964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:01.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 0x7f04b41964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57924/0 (socket says 172.21.15.61:57924) 2023-07-20T19:05:01.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 -- 172.21.15.61:0/3525442474 learned_addr learned my addr 172.21.15.61:0/3525442474 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:05:01.409 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b27fc700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4101a00 0x7f04b4196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b37fe700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 0x7f04b419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 -- 172.21.15.61:0/3525442474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4101a00 msgr2=0x7f04b4196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4101a00 0x7f04b4196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 -- 172.21.15.61:0/3525442474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 msgr2=0x7f04b419da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 0x7f04b419da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:01.411 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 -- 172.21.15.61:0/3525442474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0095f0 con 0x7f04b4100800 2023-07-20T19:05:01.411 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b37fe700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 0x7f04b419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:05:01.411 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b27fc700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4101a00 0x7f04b4196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:05:01.411 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.348+0000 7f04b2ffd700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 0x7f04b41964c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f04a400d9b0 tx=0x7f04a400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:05:01.412 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.349+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a400bb60 con 0x7f04b4100800 2023-07-20T19:05:01.412 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.349+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b419e600 con 0x7f04b4100800 2023-07-20T19:05:01.412 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.349+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04a4007620 con 0x7f04b4100800 2023-07-20T19:05:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.349+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a400f620 con 0x7f04b4100800 2023-07-20T19:05:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.349+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b419eb50 con 0x7f04b4100800 2023-07-20T19:05:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.350+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b410c000 con 0x7f04b4100800 2023-07-20T19:05:01.414 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.351+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f04a4010460 con 0x7f04b4100800 2023-07-20T19:05:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.351+0000 7f04b89ad700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f04a00a1150 0x7f04a00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.351+0000 7f04b27fc700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f04a00a1150 0x7f04a00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.351+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5862+0+0 (secure 0 0 0) 0x7f04a40cc180 con 0x7f04b4100800 2023-07-20T19:05:01.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.352+0000 7f04b27fc700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f04a00a1150 0x7f04a00a3610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f049c000c00 tx=0x7f049c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:05:01.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.356+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f04a4096260 con 0x7f04b4100800 2023-07-20T19:05:01.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:01.563+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f04b419f190 con 0x7f04a00a1150 2023-07-20T19:05:01.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:05:01.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:05:01.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:05:01.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:01.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626] boot 2023-07-20T19:05:01.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T19:05:01.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:01 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:05:01.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:05:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:05:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626] boot 2023-07-20T19:05:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T19:05:01.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:01 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:02.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:05:02.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:05:02.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T19:05:02.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:02.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: osd.8 [v2:172.21.15.156:6816/1602221626,v1:172.21.15.156:6817/1602221626] boot 2023-07-20T19:05:02.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T19:05:02.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:01 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T19:05:02.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:02 smithi061 ceph-mon[127454]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T19:05:02.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:02 smithi061 ceph-mon[127454]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:02.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:02.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:02.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:02.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:02 smithi156 ceph-mon[129228]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T19:05:02.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:02 smithi156 ceph-mon[129228]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:02.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:02.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:02.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:03.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:02 smithi149 ceph-mon[128885]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T19:05:03.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:02 smithi149 ceph-mon[128885]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi061:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:03.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:03.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:03.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:03.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:03 smithi156 ceph-mon[129228]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:03.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:03 smithi156 ceph-mon[129228]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T19:05:03.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:03 smithi061 ceph-mon[127454]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:03.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:03 smithi061 ceph-mon[127454]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T19:05:04.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:03 smithi149 ceph-mon[128885]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:04.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:03 smithi149 ceph-mon[128885]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T19:05:04.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:04 smithi156 ceph-mon[129228]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T19:05:04.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:04 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1551105493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac0b69d5-ccea-470d-a8c8-b9a2f508af24"}]: dispatch 2023-07-20T19:05:04.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:04 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1551105493' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac0b69d5-ccea-470d-a8c8-b9a2f508af24"}]': finished 2023-07-20T19:05:04.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:04 smithi156 ceph-mon[129228]: osdmap e61: 10 total, 9 up, 10 in 2023-07-20T19:05:04.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:04 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:04.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:04 smithi061 ceph-mon[127454]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T19:05:04.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:04 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1551105493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac0b69d5-ccea-470d-a8c8-b9a2f508af24"}]: dispatch 2023-07-20T19:05:04.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:04 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1551105493' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac0b69d5-ccea-470d-a8c8-b9a2f508af24"}]': finished 2023-07-20T19:05:04.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:04 smithi061 ceph-mon[127454]: osdmap e61: 10 total, 9 up, 10 in 2023-07-20T19:05:04.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:04 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:05.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:04 smithi149 ceph-mon[128885]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T19:05:05.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:04 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1551105493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac0b69d5-ccea-470d-a8c8-b9a2f508af24"}]: dispatch 2023-07-20T19:05:05.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:04 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1551105493' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac0b69d5-ccea-470d-a8c8-b9a2f508af24"}]': finished 2023-07-20T19:05:05.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:04 smithi149 ceph-mon[128885]: osdmap e61: 10 total, 9 up, 10 in 2023-07-20T19:05:05.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:04 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:05.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:05 smithi156 ceph-mon[129228]: pgmap v178: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:05.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:05 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/4130968483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:05.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:05 smithi061 ceph-mon[127454]: pgmap v178: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:05.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:05 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/4130968483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:06.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:05 smithi149 ceph-mon[128885]: pgmap v178: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:06.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:05 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/4130968483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:07.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:07 smithi156 ceph-mon[129228]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2023-07-20T19:05:07.968 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:07 smithi061 ceph-mon[127454]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2023-07-20T19:05:08.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:07 smithi149 ceph-mon[128885]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2023-07-20T19:05:09.746 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:09 smithi061 ceph-mon[127454]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:05:09.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:09 smithi156 ceph-mon[129228]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:05:10.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:09 smithi149 ceph-mon[128885]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:05:10.718 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T19:05:10.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:10 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:10.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T19:05:10.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:10 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:11.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T19:05:11.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:10 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:11.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:11 smithi061 ceph-mon[127454]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T19:05:11.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:11 smithi061 ceph-mon[127454]: Deploying daemon osd.9 on smithi061 2023-07-20T19:05:11.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:11 smithi156 ceph-mon[129228]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T19:05:11.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:11 smithi156 ceph-mon[129228]: Deploying daemon osd.9 on smithi061 2023-07-20T19:05:12.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:11 smithi149 ceph-mon[128885]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T19:05:12.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:11 smithi149 ceph-mon[128885]: Deploying daemon osd.9 on smithi061 2023-07-20T19:05:13.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:13 smithi156 ceph-mon[129228]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T19:05:13.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:13 smithi061 ceph-mon[127454]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T19:05:14.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:13 smithi149 ceph-mon[128885]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T19:05:15.907 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:15 smithi061 ceph-mon[127454]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:05:15.907 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:15.908 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:15.908 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:15.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:15 smithi156 ceph-mon[129228]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:05:15.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:15.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:15.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:16.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:15 smithi149 ceph-mon[128885]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:05:16.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:16.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:16.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:17.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.410+0000 7f04b89ad700 1 -- 172.21.15.61:0/3525442474 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f04b419f190 con 0x7f04a00a1150 2023-07-20T19:05:17.470 INFO:teuthology.orchestra.run.smithi061.stdout:Created osd(s) 9 on host 'smithi061' 2023-07-20T19:05:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f04a00a1150 msgr2=0x7f04a00a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f04a00a1150 0x7f04a00a3610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f049c000c00 tx=0x7f049c019040 comp rx=0 tx=0).stop 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 msgr2=0x7f04b41964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 0x7f04b41964c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f04a400d9b0 tx=0x7f04a400dd70 comp rx=0 tx=0).stop 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 shutdown_connections 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f04a00a1150 0x7f04a00a3610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f04b41023a0 0x7f04b419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04b4100800 0x7f04b41964c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 --2- 172.21.15.61:0/3525442474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04b4101a00 0x7f04b4196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 >> 172.21.15.61:0/3525442474 conn(0x7f04b40fbd60 msgr2=0x7f04b40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:05:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 shutdown_connections 2023-07-20T19:05:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:05:17.413+0000 7f04b99af700 1 -- 172.21.15.61:0/3525442474 wait complete. 2023-07-20T19:05:17.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:17 smithi061 ceph-mon[127454]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-20T19:05:17.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.894 DEBUG:teuthology.orchestra.run.smithi061:osd.9> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T19:05:17.897 INFO:tasks.cephadm:Deploying osd.10 on smithi149 with /dev/vg_nvme/lv_1... 2023-07-20T19:05:17.897 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T19:05:17.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:17 smithi156 ceph-mon[129228]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-20T19:05:17.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.927 INFO:journalctl@ceph.osd.9.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:05:17.930 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:17 smithi149 ceph-mon[128885]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-20T19:05:17.930 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.931 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.931 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:17.931 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:18.467 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:05:19.136 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:05:18 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:05:18.807+0000 7f34caaf5640 -1 osd.9 0 log_to_monitors true 2023-07-20T19:05:19.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:19 smithi156 ceph-mon[129228]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:19.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:19 smithi156 ceph-mon[129228]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T19:05:19.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:19 smithi061 ceph-mon[127454]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:19.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:19 smithi061 ceph-mon[127454]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T19:05:20.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:19 smithi149 ceph-mon[128885]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:20.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:19 smithi149 ceph-mon[128885]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T19:05:20.084 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T19:05:20.085 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T19:05:20.085 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2023-07-20T19:05:20.085 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2023-07-20T19:05:20.085 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111069 s, 944 MB/s 2023-07-20T19:05:20.085 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2023-07-20T19:05:20.520 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi149:vg_nvme/lv_1 2023-07-20T19:05:20.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:20.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:20.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:20 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:20.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:20.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:20 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.969 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:05:20.569+0000 7f34c016d700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T19:05:20.969 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:05:20.585+0000 7f34b7f5b700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:05:20.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T19:05:20.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T19:05:20.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:20.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]: dispatch 2023-07-20T19:05:20.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:20.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:20 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:21.116 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T19:05:21.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:21.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi061 2023-07-20T19:05:21.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi061 to 4292M 2023-07-20T19:05:21.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:05:21.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:21.840 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:21.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi061 2023-07-20T19:05:21.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi061 to 4292M 2023-07-20T19:05:21.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:05:21.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T19:05:21.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:21.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:21.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:21.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:21.920 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:21.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi061 2023-07-20T19:05:21.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi061 to 4292M 2023-07-20T19:05:21.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi061", "root=default"]}]': finished 2023-07-20T19:05:21.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T19:05:21.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:21.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:21.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:21.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:21.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:21.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:22.357 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 -- 172.21.15.149:0/4171624658 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 msgr2=0x7f9c50100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:22.358 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 --2- 172.21.15.149:0/4171624658 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c50100d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c440099c0 tx=0x7f9c44009cd0 comp rx=0 tx=0).stop 2023-07-20T19:05:22.358 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 -- 172.21.15.149:0/4171624658 shutdown_connections 2023-07-20T19:05:22.358 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 --2- 172.21.15.149:0/4171624658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c501012a0 0x7f9c50107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:22.358 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 --2- 172.21.15.149:0/4171624658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c500ff700 0x7f9c500ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:22.358 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 --2- 172.21.15.149:0/4171624658 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c50100d60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:22.359 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 -- 172.21.15.149:0/4171624658 >> 172.21.15.149:0/4171624658 conn(0x7f9c500fac80 msgr2=0x7f9c500fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:05:22.359 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.298+0000 7f9c569ba700 1 -- 172.21.15.149:0/4171624658 shutdown_connections 2023-07-20T19:05:22.359 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 -- 172.21.15.149:0/4171624658 wait complete. 2023-07-20T19:05:22.359 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 Processor -- start 2023-07-20T19:05:22.359 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 -- start start 2023-07-20T19:05:22.359 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c500ff700 0x7f9c50069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:22.361 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c5019b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:22.361 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c501012a0 0x7f9c5019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:22.361 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5019e000 con 0x7f9c501012a0 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5019e170 con 0x7f9c50100900 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.299+0000 7f9c569ba700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50194ff0 con 0x7f9c500ff700 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c5019b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c5019b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:44386/0 (socket says 172.21.15.149:44386) 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 -- 172.21.15.149:0/3640307217 learned_addr learned my addr 172.21.15.149:0/3640307217 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c54f57700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c501012a0 0x7f9c5019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4ffff700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c500ff700 0x7f9c50069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 -- 172.21.15.149:0/3640307217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c500ff700 msgr2=0x7f9c50069620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c500ff700 0x7f9c50069620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 -- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c501012a0 msgr2=0x7f9c5019d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c501012a0 0x7f9c5019d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 -- 172.21.15.149:0/3640307217 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c440095f0 con 0x7f9c50100900 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4f7fe700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c5019b220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009990 tx=0x7f9c44004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:05:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mon.2 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4401d070 con 0x7f9c50100900 2023-07-20T19:05:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mon.2 v2:172.21.15.149:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c44004010 con 0x7f9c50100900 2023-07-20T19:05:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c50195270 con 0x7f9c50100900 2023-07-20T19:05:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mon.2 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44022d40 con 0x7f9c50100900 2023-07-20T19:05:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.300+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c50195760 con 0x7f9c50100900 2023-07-20T19:05:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.301+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5010af00 con 0x7f9c50100900 2023-07-20T19:05:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.302+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mon.2 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9c44022470 con 0x7f9c50100900 2023-07-20T19:05:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.303+0000 7f9c4d7fa700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f9c380a11d0 0x7f9c380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:22.366 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.303+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mon.2 v2:172.21.15.149:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f9c440cd730 con 0x7f9c50100900 2023-07-20T19:05:22.366 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.304+0000 7f9c4ffff700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f9c380a11d0 0x7f9c380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:22.366 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.304+0000 7f9c4ffff700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f9c380a11d0 0x7f9c380a3690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c50100760 tx=0x7f9c4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:05:22.366 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.307+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mon.2 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9c44097710 con 0x7f9c50100900 2023-07-20T19:05:22.560 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:22.501+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c501962f0 con 0x7f9c380a11d0 2023-07-20T19:05:22.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:05:22.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:05:22.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682] boot 2023-07-20T19:05:22.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: osdmap e64: 10 total, 10 up, 10 in 2023-07-20T19:05:22.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:22.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:22.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:22.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:22.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:05:22.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:05:22.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682] boot 2023-07-20T19:05:22.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: osdmap e64: 10 total, 10 up, 10 in 2023-07-20T19:05:22.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:22.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:22.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:22.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:23.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:05:23.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:05:23.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: osd.9 [v2:172.21.15.61:6826/2654664682,v1:172.21.15.61:6827/2654664682] boot 2023-07-20T19:05:23.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: osdmap e64: 10 total, 10 up, 10 in 2023-07-20T19:05:23.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T19:05:23.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:23.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:23.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:23.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:23 smithi156 ceph-mon[129228]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:23.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:23 smithi156 ceph-mon[129228]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:23.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:23 smithi156 ceph-mon[129228]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T19:05:23.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:23 smithi061 ceph-mon[127454]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:23.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:23 smithi061 ceph-mon[127454]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:23.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:23 smithi061 ceph-mon[127454]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T19:05:24.004 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:23 smithi149 ceph-mon[128885]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T19:05:24.004 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:23 smithi149 ceph-mon[128885]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:24.004 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:23 smithi149 ceph-mon[128885]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T19:05:25.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:25 smithi156 ceph-mon[129228]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:25.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:25 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/1044773354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]: dispatch 2023-07-20T19:05:25.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:25 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]: dispatch 2023-07-20T19:05:25.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:25 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]': finished 2023-07-20T19:05:25.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:25 smithi156 ceph-mon[129228]: osdmap e66: 11 total, 10 up, 11 in 2023-07-20T19:05:25.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:25 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:25.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:25 smithi061 ceph-mon[127454]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:25.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:25 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/1044773354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]: dispatch 2023-07-20T19:05:25.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:25 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]: dispatch 2023-07-20T19:05:25.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:25 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]': finished 2023-07-20T19:05:25.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:25 smithi061 ceph-mon[127454]: osdmap e66: 11 total, 10 up, 11 in 2023-07-20T19:05:25.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:25 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:26.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:25 smithi149 ceph-mon[128885]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:26.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:25 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/1044773354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]: dispatch 2023-07-20T19:05:26.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:25 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]: dispatch 2023-07-20T19:05:26.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:25 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a6e7d98-a3f2-4d81-843e-bca056ec748c"}]': finished 2023-07-20T19:05:26.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:25 smithi149 ceph-mon[128885]: osdmap e66: 11 total, 10 up, 11 in 2023-07-20T19:05:26.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:25 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:26.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:26 smithi061 ceph-mon[127454]: from='client.? 172.21.15.149:0/2498406169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:27.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:26 smithi149 ceph-mon[128885]: from='client.? 172.21.15.149:0/2498406169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:27.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:26 smithi156 ceph-mon[129228]: from='client.? 172.21.15.149:0/2498406169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:27.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:27 smithi061 ceph-mon[127454]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:28.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:27 smithi149 ceph-mon[128885]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:28.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:27 smithi156 ceph-mon[129228]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:29.928 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:29 smithi149 ceph-mon[128885]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:29.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:29 smithi061 ceph-mon[127454]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:30.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:29 smithi156 ceph-mon[129228]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:31.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:31 smithi149 ceph-mon[128885]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:31.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:31 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T19:05:31.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:31 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:31.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:31 smithi061 ceph-mon[127454]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:31.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:31 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T19:05:31.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:31 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:32.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:31 smithi156 ceph-mon[129228]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:32.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:31 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T19:05:32.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:31 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:32.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:32 smithi061 ceph-mon[127454]: Deploying daemon osd.10 on smithi149 2023-07-20T19:05:33.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:32 smithi149 ceph-mon[128885]: Deploying daemon osd.10 on smithi149 2023-07-20T19:05:33.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:32 smithi156 ceph-mon[129228]: Deploying daemon osd.10 on smithi149 2023-07-20T19:05:33.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:33 smithi149 ceph-mon[128885]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:33.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:33 smithi061 ceph-mon[127454]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:34.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:33 smithi156 ceph-mon[129228]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:35.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:35 smithi061 ceph-mon[127454]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:35.971 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:35 smithi149 ceph-mon[128885]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:36.166 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:35 smithi156 ceph-mon[129228]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:37.303 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:37 smithi149 ceph-mon[128885]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:37.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:37.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:37.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:37.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:37 smithi061 ceph-mon[127454]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:37.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:37.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:37.469 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:37.666 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:37 smithi156 ceph-mon[129228]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:37.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:37.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:37.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:39.234 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 10 on host 'smithi149' 2023-07-20T19:05:39.235 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.176+0000 7f9c4d7fa700 1 -- 172.21.15.149:0/3640307217 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9c501962f0 con 0x7f9c380a11d0 2023-07-20T19:05:39.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.178+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f9c380a11d0 msgr2=0x7f9c380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:39.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.178+0000 7f9c569ba700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f9c380a11d0 0x7f9c380a3690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c50100760 tx=0x7f9c4000b410 comp rx=0 tx=0).stop 2023-07-20T19:05:39.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.178+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 msgr2=0x7f9c5019b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:39.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.178+0000 7f9c569ba700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c5019b220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009990 tx=0x7f9c44004730 comp rx=0 tx=0).stop 2023-07-20T19:05:39.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 shutdown_connections 2023-07-20T19:05:39.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f9c380a11d0 0x7f9c380a3690 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:39.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c501012a0 0x7f9c5019d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:39.238 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c500ff700 0x7f9c50069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:39.238 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 --2- 172.21.15.149:0/3640307217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c50100900 0x7f9c5019b220 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:39.238 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 >> 172.21.15.149:0/3640307217 conn(0x7f9c500fac80 msgr2=0x7f9c500fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:05:39.238 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 shutdown_connections 2023-07-20T19:05:39.238 INFO:teuthology.orchestra.run.smithi149.stderr:2023-07-20T19:05:39.179+0000 7f9c569ba700 1 -- 172.21.15.149:0/3640307217 wait complete. 2023-07-20T19:05:39.707 DEBUG:teuthology.orchestra.run.smithi149:osd.10> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T19:05:39.710 INFO:tasks.cephadm:Deploying osd.11 on smithi156 with /dev/vg_nvme/lv_1... 2023-07-20T19:05:39.711 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T19:05:39.740 INFO:journalctl@ceph.osd.10.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:05:39.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:39 smithi156 ceph-mon[129228]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:39.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:39 smithi061 ceph-mon[127454]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:40.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:39 smithi149 ceph-mon[128885]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:40.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:40.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:40.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:40.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:40.278 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:05:40.449 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:05:40 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:05:40.046+0000 7f1ac2187640 -1 osd.10 0 log_to_monitors true 2023-07-20T19:05:40.778 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:40 smithi156 ceph-mon[129228]: from='osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T19:05:40.778 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:40 smithi156 ceph-mon[129228]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T19:05:40.803 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:40 smithi149 ceph-mon[128885]: from='osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T19:05:40.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:40 smithi149 ceph-mon[128885]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T19:05:40.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:40 smithi061 ceph-mon[127454]: from='osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T19:05:40.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:40 smithi061 ceph-mon[127454]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T19:05:41.773 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T19:05:41.773 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T19:05:41.773 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-20T19:05:41.773 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-20T19:05:41.774 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00807947 s, 1.3 GB/s 2023-07-20T19:05:41.774 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-20T19:05:41.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:41 smithi156 ceph-mon[129228]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:41.912 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:41 smithi156 ceph-mon[129228]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T19:05:41.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:41 smithi156 ceph-mon[129228]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T19:05:41.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:41.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:41 smithi156 ceph-mon[129228]: from='osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:05:41.913 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:41 smithi156 ceph-mon[129228]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:05:41.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:41 smithi061 ceph-mon[127454]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:41.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:41 smithi061 ceph-mon[127454]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T19:05:41.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:41 smithi061 ceph-mon[127454]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T19:05:41.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:41.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:41 smithi061 ceph-mon[127454]: from='osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:05:41.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:41 smithi061 ceph-mon[127454]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:05:42.034 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:05:41.524+0000 7f1ab77ff700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T19:05:42.034 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:05:41.536+0000 7f1ab35f5700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:05:42.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-mon[128885]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:42.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-mon[128885]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T19:05:42.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-mon[128885]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T19:05:42.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:42.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-mon[128885]: from='osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:05:42.035 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:41 smithi149 ceph-mon[128885]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2023-07-20T19:05:42.136 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2023-07-20T19:05:42.690 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:05:42.885 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:05:42.885 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T19:05:42.885 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:42.885 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:42.885 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:42.886 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375] boot 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: osdmap e69: 11 total, 11 up, 11 in 2023-07-20T19:05:42.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:42.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:05:42.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:42.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:42.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:42.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:42.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375] boot 2023-07-20T19:05:42.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: osdmap e69: 11 total, 11 up, 11 in 2023-07-20T19:05:42.972 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:43.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:43.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:43.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:05:43.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:43.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: osd.10 [v2:172.21.15.149:6824/1912645375,v1:172.21.15.149:6825/1912645375] boot 2023-07-20T19:05:43.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: osdmap e69: 11 total, 11 up, 11 in 2023-07-20T19:05:43.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T19:05:43.782 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3898600299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 msgr2=0x7f0d18107c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:43.783 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3898600299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18107c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800b0d0 tx=0x7f0d0800b3e0 comp rx=0 tx=0).stop 2023-07-20T19:05:43.783 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3898600299 shutdown_connections 2023-07-20T19:05:43.783 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3898600299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18107c60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:43.784 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3898600299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d18100d50 0x7f0d18101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:43.784 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3898600299 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d180ffa00 0x7f0d180ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:43.784 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3898600299 >> 172.21.15.156:0/3898600299 conn(0x7f0d180fa450 msgr2=0x7f0d180fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:05:43.784 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3898600299 shutdown_connections 2023-07-20T19:05:43.785 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.723+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3898600299 wait complete. 2023-07-20T19:05:43.788 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 Processor -- start 2023-07-20T19:05:43.788 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 -- start start 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d180ffa00 0x7f0d18195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d18100d50 0x7f0d1819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1819d0e0 con 0x7f0d180ffa00 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1819d220 con 0x7f0d18100d50 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1cebf700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1819d3b0 con 0x7f0d181003c0 2023-07-20T19:05:43.789 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d15d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d15d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48732/0 (socket says 172.21.15.156:48732) 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d15d9b700 1 -- 172.21.15.156:0/3750173516 learned_addr learned my addr 172.21.15.156:0/3750173516 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d1659c700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d180ffa00 0x7f0d18195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.724+0000 7f0d16d9d700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d18100d50 0x7f0d1819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d15d9b700 1 -- 172.21.15.156:0/3750173516 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d18100d50 msgr2=0x7f0d1819cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d15d9b700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d18100d50 0x7f0d1819cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:43.790 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d15d9b700 1 -- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d180ffa00 msgr2=0x7f0d18195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d15d9b700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d180ffa00 0x7f0d18195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d15d9b700 1 -- 172.21.15.156:0/3750173516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08009d00 con 0x7f0d181003c0 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d16d9d700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d18100d50 0x7f0d1819cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d15d9b700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18195a90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00b530 tx=0x7f0d0c00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c011830 con 0x7f0d181003c0 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1819d690 con 0x7f0d181003c0 2023-07-20T19:05:43.791 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1819dc60 con 0x7f0d181003c0 2023-07-20T19:05:43.792 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d0c011e50 con 0x7f0d181003c0 2023-07-20T19:05:43.792 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.725+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00f2e0 con 0x7f0d181003c0 2023-07-20T19:05:43.792 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.726+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d18106d10 con 0x7f0d181003c0 2023-07-20T19:05:43.792 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.727+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0d0c010580 con 0x7f0d181003c0 2023-07-20T19:05:43.793 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.727+0000 7f0d077fe700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0d000a11d0 0x7f0d000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:05:43.793 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.728+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f0d0c0cbdd0 con 0x7f0d181003c0 2023-07-20T19:05:43.793 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.728+0000 7f0d1659c700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0d000a11d0 0x7f0d000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:05:43.793 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.728+0000 7f0d1659c700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0d000a11d0 0x7f0d000a3690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0d18068970 tx=0x7f0d1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:05:43.794 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.732+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d0c0d0070 con 0x7f0d181003c0 2023-07-20T19:05:43.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:43 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:05:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:43 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:05:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:43 smithi156 ceph-mon[129228]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:43 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi149 2023-07-20T19:05:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:43 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi149 to 1469M 2023-07-20T19:05:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:43 smithi156 ceph-mon[129228]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T19:05:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:43 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:05:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:43 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:05:43.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:43 smithi061 ceph-mon[127454]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:43 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi149 2023-07-20T19:05:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:43 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi149 to 1469M 2023-07-20T19:05:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:43 smithi061 ceph-mon[127454]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T19:05:43.986 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:43.927+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d18060bb0 con 0x7f0d000a11d0 2023-07-20T19:05:44.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:43 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:05:44.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:43 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:05:44.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:43 smithi149 ceph-mon[128885]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T19:05:44.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:43 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi149 2023-07-20T19:05:44.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:43 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi149 to 1469M 2023-07-20T19:05:44.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:43 smithi149 ceph-mon[128885]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T19:05:44.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:44.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:44.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:44 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:44.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:44 smithi156 ceph-mon[129228]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T19:05:44.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:44.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:44.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:44 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:44.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:44 smithi061 ceph-mon[127454]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T19:05:45.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T19:05:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T19:05:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:44 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:44 smithi149 ceph-mon[128885]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T19:05:45.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:45 smithi156 ceph-mon[129228]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:05:45.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:45 smithi156 ceph-mon[129228]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:45.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:45 smithi061 ceph-mon[127454]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:05:45.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:45 smithi061 ceph-mon[127454]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:46.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:45 smithi149 ceph-mon[128885]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:05:46.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:45 smithi149 ceph-mon[128885]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:05:46.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:46 smithi156 ceph-mon[129228]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T19:05:46.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:46 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/617223524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]: dispatch 2023-07-20T19:05:46.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:46 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]: dispatch 2023-07-20T19:05:46.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:46 smithi156 ceph-mon[129228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]': finished 2023-07-20T19:05:46.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:46 smithi156 ceph-mon[129228]: osdmap e73: 12 total, 11 up, 12 in 2023-07-20T19:05:46.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:46 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:05:46.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:46 smithi061 ceph-mon[127454]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T19:05:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:46 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/617223524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]: dispatch 2023-07-20T19:05:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:46 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]: dispatch 2023-07-20T19:05:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:46 smithi061 ceph-mon[127454]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]': finished 2023-07-20T19:05:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:46 smithi061 ceph-mon[127454]: osdmap e73: 12 total, 11 up, 12 in 2023-07-20T19:05:46.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:46 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:05:47.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:46 smithi149 ceph-mon[128885]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T19:05:47.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:46 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/617223524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]: dispatch 2023-07-20T19:05:47.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:46 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]: dispatch 2023-07-20T19:05:47.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:46 smithi149 ceph-mon[128885]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bcb26b2-0789-4cf8-9803-98a6da33d5d7"}]': finished 2023-07-20T19:05:47.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:46 smithi149 ceph-mon[128885]: osdmap e73: 12 total, 11 up, 12 in 2023-07-20T19:05:47.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:46 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:05:47.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:47 smithi156 ceph-mon[129228]: pgmap v211: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T19:05:47.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:47 smithi156 ceph-mon[129228]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-20T19:05:47.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:47 smithi156 ceph-mon[129228]: from='client.? 172.21.15.156:0/434468206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:47.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:47 smithi061 ceph-mon[127454]: pgmap v211: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T19:05:47.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:47 smithi061 ceph-mon[127454]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-20T19:05:47.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:47 smithi061 ceph-mon[127454]: from='client.? 172.21.15.156:0/434468206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:48.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:47 smithi149 ceph-mon[128885]: pgmap v211: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T19:05:48.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:47 smithi149 ceph-mon[128885]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-20T19:05:48.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:47 smithi149 ceph-mon[128885]: from='client.? 172.21.15.156:0/434468206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T19:05:49.916 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:49 smithi156 ceph-mon[129228]: pgmap v213: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T19:05:49.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:49 smithi061 ceph-mon[127454]: pgmap v213: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T19:05:50.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:49 smithi149 ceph-mon[128885]: pgmap v213: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-20T19:05:50.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:50 smithi061 ceph-mon[127454]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-20T19:05:50.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:50 smithi061 ceph-mon[127454]: Cluster is now healthy 2023-07-20T19:05:51.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:50 smithi149 ceph-mon[128885]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-20T19:05:51.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:50 smithi149 ceph-mon[128885]: Cluster is now healthy 2023-07-20T19:05:51.116 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:50 smithi156 ceph-mon[129228]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-20T19:05:51.116 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:50 smithi156 ceph-mon[129228]: Cluster is now healthy 2023-07-20T19:05:51.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:51 smithi156 ceph-mon[129228]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:05:51.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:51 smithi061 ceph-mon[127454]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:05:52.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:51 smithi149 ceph-mon[128885]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T19:05:53.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:53 smithi156 ceph-mon[129228]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T19:05:53.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T19:05:53.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:53 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:53.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:53 smithi156 ceph-mon[129228]: Deploying daemon osd.11 on smithi156 2023-07-20T19:05:53.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:53 smithi061 ceph-mon[127454]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T19:05:53.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T19:05:53.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:53 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:53.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:53 smithi061 ceph-mon[127454]: Deploying daemon osd.11 on smithi156 2023-07-20T19:05:54.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:53 smithi149 ceph-mon[128885]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-20T19:05:54.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T19:05:54.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:53 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:05:54.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:53 smithi149 ceph-mon[128885]: Deploying daemon osd.11 on smithi156 2023-07-20T19:05:55.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:55 smithi156 ceph-mon[129228]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T19:05:55.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:55 smithi061 ceph-mon[127454]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T19:05:56.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:55 smithi149 ceph-mon[128885]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-20T19:05:57.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:57 smithi156 ceph-mon[129228]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:05:57.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:57.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:57.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:57 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:57.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:57 smithi061 ceph-mon[127454]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:05:57.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:57.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:57.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:57 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:58.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:57 smithi149 ceph-mon[128885]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T19:05:58.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:58.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:05:58.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:57 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:05:59.783 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:05:59 smithi156 ceph-mon[129228]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T19:05:59.849 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.788+0000 7f0d077fe700 1 -- 172.21.15.156:0/3750173516 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f0d18060bb0 con 0x7f0d000a11d0 2023-07-20T19:05:59.850 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 11 on host 'smithi156' 2023-07-20T19:05:59.851 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.790+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0d000a11d0 msgr2=0x7f0d000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:59.851 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.790+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0d000a11d0 0x7f0d000a3690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0d18068970 tx=0x7f0d1000b410 comp rx=0 tx=0).stop 2023-07-20T19:05:59.851 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.790+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 msgr2=0x7f0d18195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:05:59.851 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.790+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18195a90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00b530 tx=0x7f0d0c00b840 comp rx=0 tx=0).stop 2023-07-20T19:05:59.851 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 shutdown_connections 2023-07-20T19:05:59.851 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0d000a11d0 0x7f0d000a3690 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:59.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0d18100d50 0x7f0d1819cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:59.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d180ffa00 0x7f0d18195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:59.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 --2- 172.21.15.156:0/3750173516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d181003c0 0x7f0d18195a90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:05:59.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 >> 172.21.15.156:0/3750173516 conn(0x7f0d180fa450 msgr2=0x7f0d18103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:05:59.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 shutdown_connections 2023-07-20T19:05:59.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:05:59.791+0000 7f0d1cebf700 1 -- 172.21.15.156:0/3750173516 wait complete. 2023-07-20T19:05:59.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:05:59 smithi061 ceph-mon[127454]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T19:06:00.053 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:05:59 smithi149 ceph-mon[128885]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-20T19:06:00.286 DEBUG:teuthology.orchestra.run.smithi156:osd.11> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T19:06:00.288 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T19:06:00.288 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T19:06:00.320 INFO:journalctl@ceph.osd.11.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:06:00.866 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:06:01.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:00 smithi149 ceph-mon[128885]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T19:06:01.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:00 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.057 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:00 smithi061 ceph-mon[127454]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T19:06:01.057 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.058 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.058 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.058 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:00 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.067 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:00 smithi156 ceph-mon[129228]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T19:06:01.067 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.067 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.067 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.067 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:00 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:01.417 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:06:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:06:01.007+0000 7f3035302640 -1 osd.11 0 log_to_monitors true 2023-07-20T19:06:01.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:01 smithi156 ceph-mon[129228]: from='osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T19:06:01.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:01 smithi156 ceph-mon[129228]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T19:06:01.935 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 -- 172.21.15.61:0/3020026361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 msgr2=0x7f0b7c100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:01.935 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 --2- 172.21.15.61:0/3020026361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c100c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0099c0 tx=0x7f0b6c009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 -- 172.21.15.61:0/3020026361 shutdown_connections 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 --2- 172.21.15.61:0/3020026361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b7c1023d0 0x7f0b7c108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 --2- 172.21.15.61:0/3020026361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c100c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 --2- 172.21.15.61:0/3020026361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c101a30 0x7f0b7c101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 -- 172.21.15.61:0/3020026361 >> 172.21.15.61:0/3020026361 conn(0x7f0b7c0fbdf0 msgr2=0x7f0b7c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 -- 172.21.15.61:0/3020026361 shutdown_connections 2023-07-20T19:06:01.936 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.874+0000 7f0b825af700 1 -- 172.21.15.61:0/3020026361 wait complete. 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 Processor -- start 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 -- start start 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b7c101a30 0x7f0b7c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c1023d0 0x7f0b7c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19e080 con 0x7f0b7c100830 2023-07-20T19:06:01.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19e1c0 con 0x7f0b7c1023d0 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b825af700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19e350 con 0x7f0b7c101a30 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c1964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58084/0 (socket says 172.21.15.61:58084) 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b7bfff700 1 -- 172.21.15.61:0/4214273921 learned_addr learned my addr 172.21.15.61:0/4214273921 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b7b7fe700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b7c101a30 0x7f0b7c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.875+0000 7f0b80b4c700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c1023d0 0x7f0b7c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b7bfff700 1 -- 172.21.15.61:0/4214273921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b7c101a30 msgr2=0x7f0b7c196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:01.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b7bfff700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b7c101a30 0x7f0b7c196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b7bfff700 1 -- 172.21.15.61:0/4214273921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c1023d0 msgr2=0x7f0b7c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b7bfff700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c1023d0 0x7f0b7c19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b7bfff700 1 -- 172.21.15.61:0/4214273921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0095f0 con 0x7f0b7c100830 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b80b4c700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c1023d0 0x7f0b7c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b7bfff700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c1964f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0036f0 tx=0x7f0b6c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c01d070 con 0x7f0b7c100830 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c0729a0 con 0x7f0b7c100830 2023-07-20T19:06:01.939 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b6c004010 con 0x7f0b7c100830 2023-07-20T19:06:01.940 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c022d70 con 0x7f0b7c100830 2023-07-20T19:06:01.940 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.876+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c072e90 con 0x7f0b7c100830 2023-07-20T19:06:01.940 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.877+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c107e40 con 0x7f0b7c100830 2023-07-20T19:06:01.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.878+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0b6c022470 con 0x7f0b7c100830 2023-07-20T19:06:01.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.878+0000 7f0b797fa700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0b680a11d0 0x7f0b680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:01.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.879+0000 7f0b7b7fe700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0b680a11d0 0x7f0b680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:01.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.879+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f0b6c0cdea0 con 0x7f0b7c100830 2023-07-20T19:06:01.942 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.879+0000 7f0b7b7fe700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0b680a11d0 0x7f0b680a3690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0b64005950 tx=0x7f0b6400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:01.942 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:01.882+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0b6c097bb0 con 0x7f0b7c100830 2023-07-20T19:06:02.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:01 smithi149 ceph-mon[128885]: from='osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T19:06:02.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:01 smithi149 ceph-mon[128885]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T19:06:02.134 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.076+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0b7c02c970 con 0x7f0b7c100830 2023-07-20T19:06:02.134 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.076+0000 7f0b797fa700 1 -- 172.21.15.61:0/4214273921 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7f0b6c09b1d0 con 0x7f0b7c100830 2023-07-20T19:06:02.134 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:02.136 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0b680a11d0 msgr2=0x7f0b680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:02.136 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0b680a11d0 0x7f0b680a3690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0b64005950 tx=0x7f0b6400b410 comp rx=0 tx=0).stop 2023-07-20T19:06:02.136 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 msgr2=0x7f0b7c1964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c1964f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0036f0 tx=0x7f0b6c005720 comp rx=0 tx=0).stop 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 shutdown_connections 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0b680a11d0 0x7f0b680a3690 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b7c1023d0 0x7f0b7c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b7c100830 0x7f0b7c1964f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 --2- 172.21.15.61:0/4214273921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b7c101a30 0x7f0b7c196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 >> 172.21.15.61:0/4214273921 conn(0x7f0b7c0fbdf0 msgr2=0x7f0b7c104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:02.137 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 shutdown_connections 2023-07-20T19:06:02.138 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:02.078+0000 7f0b825af700 1 -- 172.21.15.61:0/4214273921 wait complete. 2023-07-20T19:06:02.160 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:01 smithi061 ceph-mon[127454]: from='osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T19:06:02.161 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:01 smithi061 ceph-mon[127454]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T19:06:02.529 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1689879942,"num_in_osds":12,"osd_in_since":1689879946,"num_remapped_pgs":0} 2023-07-20T19:06:03.167 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:06:02.758+0000 7f302a97a700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T19:06:03.168 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:06:02.769+0000 7f3022768700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T19:06:03.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:06:03.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T19:06:03.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T19:06:03.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:03.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:06:03.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:06:03.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/4214273921' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T19:06:03.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: Detected new or changed devices on smithi156 2023-07-20T19:06:03.170 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.171 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.171 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.171 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.172 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.172 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.172 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: Adjusting osd_memory_target on smithi156 to 4292M 2023-07-20T19:06:03.172 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.173 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:03.173 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:03.173 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.174 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:03.174 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:03.174 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:03.174 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:02 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:06:03.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T19:06:03.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T19:06:03.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:03.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/4214273921' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: Detected new or changed devices on smithi156 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: Adjusting osd_memory_target on smithi156 to 4292M 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:03.223 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:03.223 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:03.223 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:02 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:06:03.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T19:06:03.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T19:06:03.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/4214273921' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: Detected new or changed devices on smithi156 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: Adjusting osd_memory_target on smithi156 to 4292M 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:03.307 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:03.307 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:03.307 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:02 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:03.530 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T19:06:04.095 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:06:04.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:03 smithi156 ceph-mon[129228]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:06:04.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:03 smithi156 ceph-mon[129228]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T19:06:04.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:03 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:04.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:03 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:04.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:03 smithi061 ceph-mon[127454]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:06:04.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:03 smithi061 ceph-mon[127454]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T19:06:04.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:03 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:04.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:03 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:04.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:03 smithi149 ceph-mon[128885]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-20T19:06:04.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:03 smithi149 ceph-mon[128885]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T19:06:04.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:03 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:04.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:03 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:05.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:04 smithi156 ceph-mon[129228]: purged_snaps scrub starts 2023-07-20T19:06:05.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:04 smithi156 ceph-mon[129228]: purged_snaps scrub ok 2023-07-20T19:06:05.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:04 smithi156 ceph-mon[129228]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:06:05.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:04 smithi156 ceph-mon[129228]: osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887] boot 2023-07-20T19:06:05.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:04 smithi156 ceph-mon[129228]: osdmap e76: 12 total, 12 up, 12 in 2023-07-20T19:06:05.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:04 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:05.175 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 -- 172.21.15.61:0/3515887450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68041012a0 msgr2=0x7f6804107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:05.175 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 --2- 172.21.15.61:0/3515887450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68041012a0 0x7f6804107b50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00b0d0 tx=0x7f67fc00b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.175 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 -- 172.21.15.61:0/3515887450 shutdown_connections 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 --2- 172.21.15.61:0/3515887450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68041012a0 0x7f6804107b50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 --2- 172.21.15.61:0/3515887450 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68040ff700 0x7f68040ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 --2- 172.21.15.61:0/3515887450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6804100900 0x7f6804100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 -- 172.21.15.61:0/3515887450 >> 172.21.15.61:0/3515887450 conn(0x7f68040fac80 msgr2=0x7f68040fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 -- 172.21.15.61:0/3515887450 shutdown_connections 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.115+0000 7f6809ce9700 1 -- 172.21.15.61:0/3515887450 wait complete. 2023-07-20T19:06:05.176 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 Processor -- start 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 -- start start 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68040ff700 0x7f6804068df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 0x7f680419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68041012a0 0x7f680419d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680419e0b0 con 0x7f6804100900 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680419e220 con 0x7f68041012a0 2023-07-20T19:06:05.177 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6809ce9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680419e3e0 con 0x7f68040ff700 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.116+0000 7f6802ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 0x7f680419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 0x7f680419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58112/0 (socket says 172.21.15.61:58112) 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 -- 172.21.15.61:0/2511453189 learned_addr learned my addr 172.21.15.61:0/2511453189 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f68037fe700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68040ff700 0x7f6804068df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 -- 172.21.15.61:0/2511453189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68040ff700 msgr2=0x7f6804068df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6803fff700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68041012a0 0x7f680419d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68040ff700 0x7f6804068df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 -- 172.21.15.61:0/2511453189 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68041012a0 msgr2=0x7f680419d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68041012a0 0x7f680419d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 -- 172.21.15.61:0/2511453189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc009d00 con 0x7f6804100900 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6803fff700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68041012a0 0x7f680419d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6802ffd700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 0x7f680419b280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67f800e9a0 tx=0x7f67f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8009790 con 0x7f6804100900 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67f800cbd0 con 0x7f6804100900 2023-07-20T19:06:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680419e6c0 con 0x7f6804100900 2023-07-20T19:06:05.180 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8018680 con 0x7f6804100900 2023-07-20T19:06:05.180 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.117+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680419ebc0 con 0x7f6804100900 2023-07-20T19:06:05.180 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.118+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680410af10 con 0x7f6804100900 2023-07-20T19:06:05.181 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.119+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f67f800cd40 con 0x7f6804100900 2023-07-20T19:06:05.181 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.120+0000 7f6800ff9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f67ec0a1150 0x7f67ec0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:05.181 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.120+0000 7f68037fe700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f67ec0a1150 0x7f67ec0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:05.181 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.120+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f67f8014070 con 0x7f6804100900 2023-07-20T19:06:05.181 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.120+0000 7f68037fe700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f67ec0a1150 0x7f67ec0a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6804100760 tx=0x7f67f4007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:05.182 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.124+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f67f8051780 con 0x7f6804100900 2023-07-20T19:06:05.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:04 smithi061 ceph-mon[127454]: purged_snaps scrub starts 2023-07-20T19:06:05.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:04 smithi061 ceph-mon[127454]: purged_snaps scrub ok 2023-07-20T19:06:05.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:04 smithi061 ceph-mon[127454]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:06:05.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:04 smithi061 ceph-mon[127454]: osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887] boot 2023-07-20T19:06:05.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:04 smithi061 ceph-mon[127454]: osdmap e76: 12 total, 12 up, 12 in 2023-07-20T19:06:05.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:04 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:05.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:04 smithi149 ceph-mon[128885]: purged_snaps scrub starts 2023-07-20T19:06:05.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:04 smithi149 ceph-mon[128885]: purged_snaps scrub ok 2023-07-20T19:06:05.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:04 smithi149 ceph-mon[128885]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T19:06:05.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:04 smithi149 ceph-mon[128885]: osd.11 [v2:172.21.15.156:6824/893354887,v1:172.21.15.156:6825/893354887] boot 2023-07-20T19:06:05.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:04 smithi149 ceph-mon[128885]: osdmap e76: 12 total, 12 up, 12 in 2023-07-20T19:06:05.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:04 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T19:06:05.389 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.330+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f68041065a0 con 0x7f6804100900 2023-07-20T19:06:05.389 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.331+0000 7f6800ff9700 1 -- 172.21.15.61:0/2511453189 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7f67f8096250 con 0x7f6804100900 2023-07-20T19:06:05.389 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:05.391 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f67ec0a1150 msgr2=0x7f67ec0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:05.391 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f67ec0a1150 0x7f67ec0a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6804100760 tx=0x7f67f4007400 comp rx=0 tx=0).stop 2023-07-20T19:06:05.391 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 msgr2=0x7f680419b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 0x7f680419b280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67f800e9a0 tx=0x7f67f800ecb0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 shutdown_connections 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f67ec0a1150 0x7f67ec0a3610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68041012a0 0x7f680419d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68040ff700 0x7f6804068df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 --2- 172.21.15.61:0/2511453189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6804100900 0x7f680419b280 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 >> 172.21.15.61:0/2511453189 conn(0x7f68040fac80 msgr2=0x7f68040fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:05.393 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 shutdown_connections 2023-07-20T19:06:05.393 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:05.333+0000 7f6809ce9700 1 -- 172.21.15.61:0/2511453189 wait complete. 2023-07-20T19:06:05.755 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1689879963,"num_in_osds":12,"osd_in_since":1689879946,"num_remapped_pgs":0} 2023-07-20T19:06:05.756 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T19:06:06.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:05 smithi156 ceph-mon[129228]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T19:06:06.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:05 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2511453189' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T19:06:06.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:05 smithi061 ceph-mon[127454]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T19:06:06.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:05 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2511453189' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T19:06:06.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:05 smithi149 ceph-mon[128885]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T19:06:06.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:05 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2511453189' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T19:06:06.315 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:06:07.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:06 smithi156 ceph-mon[129228]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:07.219 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:06 smithi061 ceph-mon[127454]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:07.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:06 smithi149 ceph-mon[128885]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:07.523 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.464+0000 7f8b4522c700 1 -- 172.21.15.61:0/168096902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 msgr2=0x7f8b4010c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.464+0000 7f8b4522c700 1 --2- 172.21.15.61:0/168096902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b4010c510 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800b0a0 tx=0x7f8b3800b3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 -- 172.21.15.61:0/168096902 shutdown_connections 2023-07-20T19:06:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 --2- 172.21.15.61:0/168096902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b4010c510 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 --2- 172.21.15.61:0/168096902 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b40104580 0x7f8b40104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 --2- 172.21.15.61:0/168096902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b400ff6b0 0x7f8b400ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 -- 172.21.15.61:0/168096902 >> 172.21.15.61:0/168096902 conn(0x7f8b40076770 msgr2=0x7f8b40076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:07.525 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 -- 172.21.15.61:0/168096902 shutdown_connections 2023-07-20T19:06:07.525 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.465+0000 7f8b4522c700 1 -- 172.21.15.61:0/168096902 wait complete. 2023-07-20T19:06:07.525 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 Processor -- start 2023-07-20T19:06:07.526 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 -- start start 2023-07-20T19:06:07.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b400ff6b0 0x7f8b40197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:07.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b40197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:07.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b40104580 0x7f8b401a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:07.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b40198660 con 0x7f8b40100070 2023-07-20T19:06:07.527 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b401a2dd0 con 0x7f8b40104580 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b4522c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b401a2f40 con 0x7f8b400ff6b0 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b3e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b40197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b3e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b40197d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58124/0 (socket says 172.21.15.61:58124) 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b3e59c700 1 -- 172.21.15.61:0/646777000 learned_addr learned my addr 172.21.15.61:0/646777000 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.466+0000 7f8b3f59e700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b40104580 0x7f8b401a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b3e59c700 1 -- 172.21.15.61:0/646777000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b400ff6b0 msgr2=0x7f8b40197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:07.528 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b3e59c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b400ff6b0 0x7f8b40197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b3e59c700 1 -- 172.21.15.61:0/646777000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b40104580 msgr2=0x7f8b401a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b3e59c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b40104580 0x7f8b401a2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b3e59c700 1 -- 172.21.15.61:0/646777000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b38009d00 con 0x7f8b40100070 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b3e59c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b40197d50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400b530 tx=0x7f8b3400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34011830 con 0x7f8b40100070 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b34011e50 con 0x7f8b40100070 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b401a3220 con 0x7f8b40100070 2023-07-20T19:06:07.529 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3400f2e0 con 0x7f8b40100070 2023-07-20T19:06:07.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.467+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b401a3720 con 0x7f8b40100070 2023-07-20T19:06:07.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.468+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4004e470 con 0x7f8b40100070 2023-07-20T19:06:07.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.469+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8b34010380 con 0x7f8b40100070 2023-07-20T19:06:07.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.470+0000 7f8b27fff700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8b280a1150 0x7f8b280a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:07.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.470+0000 7f8b3ed9d700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8b280a1150 0x7f8b280a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:07.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.470+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b340cc570 con 0x7f8b40100070 2023-07-20T19:06:07.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.470+0000 7f8b3ed9d700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8b280a1150 0x7f8b280a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005950 tx=0x7f8b300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:07.532 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.473+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b34096160 con 0x7f8b40100070 2023-07-20T19:06:07.733 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.675+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b401a3a70 con 0x7f8b40100070 2023-07-20T19:06:07.733 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.676+0000 7f8b27fff700 1 -- 172.21.15.61:0/646777000 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18408 (secure 0 0 0) 0x7f8b34099780 con 0x7f8b40100070 2023-07-20T19:06:07.734 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:07.734 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","created":"2023-07-20T18:59:47.496551+0000","modified":"2023-07-20T19:06:04.764327+0000","last_up_change":"2023-07-20T19:06:03.755886+0000","last_in_change":"2023-07-20T19:05:46.242910+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T19:02:56.142587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfae06af-18f4-487e-ae4b-8e8c9d36ef76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6803","nonce":1072462099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6805","nonce":1072462099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6809","nonce":1072462099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6807","nonce":1072462099}]},"public_addr":"172.21.15.61:6803/1072462099","cluster_addr":"172.21.15.61:6805/1072462099","heartbeat_back_addr":"172.21.15.61:6809/1072462099","heartbeat_front_addr":"172.21.15.61:6807/1072462099","state":["exists","up"]},{"osd":1,"uuid":"d3e2b1d4-abe5-4438-9196-6ef26230a071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6801","nonce":414280084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6803","nonce":414280084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6807","nonce":414280084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6805","nonce":414280084}]},"public_addr":"172.21.15.149:6801/414280084","cluster_addr":"172.21.15.149:6803/414280084","heartbeat_back_addr":"172.21.15.149:6807/414280084","heartbeat_front_addr":"172.21.15.149:6805/414280084","state":["exists","up"]},{"osd":2,"uuid":"34a03029-87a9-49e1-b262-32be5070df0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6801","nonce":738107812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6803","nonce":738107812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6807","nonce":738107812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6805","nonce":738107812}]},"public_addr":"172.21.15.156:6801/738107812","cluster_addr":"172.21.15.156:6803/738107812","heartbeat_back_addr":"172.21.15.156:6807/738107812","heartbeat_front_addr":"172.21.15.156:6805/738107812","state":["exists","up"]},{"osd":3,"uuid":"c5711133-530f-488a-9db4-2b8a61b7ea02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6811","nonce":1371683823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6813","nonce":1371683823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6817","nonce":1371683823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6815","nonce":1371683823}]},"public_addr":"172.21.15.61:6811/1371683823","cluster_addr":"172.21.15.61:6813/1371683823","heartbeat_back_addr":"172.21.15.61:6817/1371683823","heartbeat_front_addr":"172.21.15.61:6815/1371683823","state":["exists","up"]},{"osd":4,"uuid":"6062cdd3-5bf1-458a-8454-253627db736f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6809","nonce":3513915816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6811","nonce":3513915816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6815","nonce":3513915816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6813","nonce":3513915816}]},"public_addr":"172.21.15.149:6809/3513915816","cluster_addr":"172.21.15.149:6811/3513915816","heartbeat_back_addr":"172.21.15.149:6815/3513915816","heartbeat_front_addr":"172.21.15.149:6813/3513915816","state":["exists","up"]},{"osd":5,"uuid":"1fae1b15-43cb-40d9-ad90-73af4cc96197","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6809","nonce":211953844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6811","nonce":211953844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6815","nonce":211953844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6813","nonce":211953844}]},"public_addr":"172.21.15.156:6809/211953844","cluster_addr":"172.21.15.156:6811/211953844","heartbeat_back_addr":"172.21.15.156:6815/211953844","heartbeat_front_addr":"172.21.15.156:6813/211953844","state":["exists","up"]},{"osd":6,"uuid":"84316d15-3a51-432c-9d02-57270ac09769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6819","nonce":1770665297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6821","nonce":1770665297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6825","nonce":1770665297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6823","nonce":1770665297}]},"public_addr":"172.21.15.61:6819/1770665297","cluster_addr":"172.21.15.61:6821/1770665297","heartbeat_back_addr":"172.21.15.61:6825/1770665297","heartbeat_front_addr":"172.21.15.61:6823/1770665297","state":["exists","up"]},{"osd":7,"uuid":"705c1527-9342-402c-add4-4382e005dbfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6817","nonce":1095358817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6819","nonce":1095358817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6823","nonce":1095358817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6821","nonce":1095358817}]},"public_addr":"172.21.15.149:6817/1095358817","cluster_addr":"172.21.15.149:6819/1095358817","heartbeat_back_addr":"172.21.15.149:6823/1095358817","heartbeat_front_addr":"172.21.15.149:6821/1095358817","state":["exists","up"]},{"osd":8,"uuid":"6e790866-ee13-4c27-bbd4-ea90e853de1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6817","nonce":1602221626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6819","nonce":1602221626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6823","nonce":1602221626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6821","nonce":1602221626}]},"public_addr":"172.21.15.156:6817/1602221626","cluster_addr":"172.21.15.156:6819/1602221626","heartbeat_back_addr":"172.21.15.156:6823/1602221626","heartbeat_front_addr":"172.21.15.156:6821/1602221626","state":["exists","up"]},{"osd":9,"uuid":"ac0b69d5-ccea-470d-a8c8-b9a2f508af24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6827","nonce":2654664682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6829","nonce":2654664682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6833","nonce":2654664682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6831","nonce":2654664682}]},"public_addr":"172.21.15.61:6827/2654664682","cluster_addr":"172.21.15.61:6829/2654664682","heartbeat_back_addr":"172.21.15.61:6833/2654664682","heartbeat_front_addr":"172.21.15.61:6831/2654664682","state":["exists","up"]},{"osd":10,"uuid":"1a6e7d98-a3f2-4d81-843e-bca056ec748c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6825","nonce":1912645375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6827","nonce":1912645375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6831","nonce":1912645375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6829","nonce":1912645375}]},"public_addr":"172.21.15.149:6825/1912645375","cluster_addr":"172.21.15.149:6827/1912645375","heartbeat_back_addr":"172.21.15.149:6831/1912645375","heartbeat_front_addr":"172.21.15.149:6829/1912645375","state":["exists","up"]},{"osd":11,"uuid":"5bcb26b2-0789-4cf8-9803-98a6da33d5d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6825","nonce":893354887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6827","nonce":893354887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6831","nonce":893354887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6829","nonce":893354887}]},"public_addr":"172.21.15.156:6825/893354887","cluster_addr":"172.21.15.156:6827/893354887","heartbeat_back_addr":"172.21.15.156:6831/893354887","heartbeat_front_addr":"172.21.15.156:6829/893354887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:09.841610+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:31.306806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:52.704303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:13.230276+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:34.832690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:55.534516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:16.097468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:37.618136+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:58.685044+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:19.824839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:41.039703+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:06:02.038214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.61:6801/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6800/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/3344679989":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6801/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/170145058":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/575168883":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/2333936767":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/74416212":"2023-07-21T19:00:21.461577+0000","172.21.15.61:6800/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/2426181232":"2023-07-21T19:00:21.461577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T19:06:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8b280a1150 msgr2=0x7f8b280a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8b280a1150 0x7f8b280a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005950 tx=0x7f8b300058e0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 msgr2=0x7f8b40197d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b40197d50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400b530 tx=0x7f8b3400b840 comp rx=0 tx=0).stop 2023-07-20T19:06:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 shutdown_connections 2023-07-20T19:06:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f8b280a1150 0x7f8b280a3610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b40104580 0x7f8b401a2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b400ff6b0 0x7f8b40197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 --2- 172.21.15.61:0/646777000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40100070 0x7f8b40197d50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:07.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 >> 172.21.15.61:0/646777000 conn(0x7f8b40076770 msgr2=0x7f8b400fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:07.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 shutdown_connections 2023-07-20T19:06:07.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:07.678+0000 7f8b4522c700 1 -- 172.21.15.61:0/646777000 wait complete. 2023-07-20T19:06:08.062 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T19:02:56.142587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T19:06:08.063 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T19:06:08.620 INFO:teuthology.orchestra.run.smithi061.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T19:06:08.840 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:08 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/646777000' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:08.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:08 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/646777000' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:09.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:08 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/646777000' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:09.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 -- 172.21.15.61:0/1792889601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec0ff6b0 msgr2=0x7ff3ec0ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:09.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1792889601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec0ff6b0 0x7ff3ec0ffb30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009990 tx=0x7ff3e0009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 -- 172.21.15.61:0/1792889601 shutdown_connections 2023-07-20T19:06:09.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1792889601 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3ec100070 0x7ff3ec10c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1792889601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3ec104580 0x7ff3ec104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1792889601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec0ff6b0 0x7ff3ec0ffb30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 -- 172.21.15.61:0/1792889601 >> 172.21.15.61:0/1792889601 conn(0x7ff3ec076770 msgr2=0x7ff3ec076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.652+0000 7ff3f1227700 1 -- 172.21.15.61:0/1792889601 shutdown_connections 2023-07-20T19:06:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.653+0000 7ff3f1227700 1 -- 172.21.15.61:0/1792889601 wait complete. 2023-07-20T19:06:09.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.653+0000 7ff3f1227700 1 Processor -- start 2023-07-20T19:06:09.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 -- start start 2023-07-20T19:06:09.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3ec0ff6b0 0x7ff3ec19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:09.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 0x7ff3ec19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:09.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3ec104580 0x7ff3ec196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:09.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec196da0 con 0x7ff3ec100070 2023-07-20T19:06:09.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec196f10 con 0x7ff3ec0ff6b0 2023-07-20T19:06:09.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3f1227700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1970d0 con 0x7ff3ec104580 2023-07-20T19:06:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 0x7ff3ec19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 0x7ff3ec19cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58136/0 (socket says 172.21.15.61:58136) 2023-07-20T19:06:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 -- 172.21.15.61:0/1823936763 learned_addr learned my addr 172.21.15.61:0/1823936763 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ead9d700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3ec0ff6b0 0x7ff3ec19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3eb59e700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3ec104580 0x7ff3ec196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 -- 172.21.15.61:0/1823936763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3ec104580 msgr2=0x7ff3ec196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3ec104580 0x7ff3ec196730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 -- 172.21.15.61:0/1823936763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3ec0ff6b0 msgr2=0x7ff3ec19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3ec0ff6b0 0x7ff3ec19c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 -- 172.21.15.61:0/1823936763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e00095f0 con 0x7ff3ec100070 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.654+0000 7ff3ea59c700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 0x7ff3ec19cbf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00095c0 tx=0x7ff3e000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:09.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.655+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e001d070 con 0x7ff3ec100070 2023-07-20T19:06:09.718 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.655+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec197350 con 0x7ff3ec100070 2023-07-20T19:06:09.718 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.655+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3e000fb50 con 0x7ff3ec100070 2023-07-20T19:06:09.718 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.655+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00173f0 con 0x7ff3ec100070 2023-07-20T19:06:09.718 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.655+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec1a36a0 con 0x7ff3ec100070 2023-07-20T19:06:09.718 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.656+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec04e470 con 0x7ff3ec100070 2023-07-20T19:06:09.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.657+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff3e000fcc0 con 0x7ff3ec100070 2023-07-20T19:06:09.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.657+0000 7ff3d3fff700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ff3d40a1150 0x7ff3d40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:09.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.657+0000 7ff3ead9d700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ff3d40a1150 0x7ff3d40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:09.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.657+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff3e00cd7e0 con 0x7ff3ec100070 2023-07-20T19:06:09.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.658+0000 7ff3ead9d700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ff3d40a1150 0x7ff3d40a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00a400 tx=0x7ff3dc00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:09.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.661+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff3e0097460 con 0x7ff3ec100070 2023-07-20T19:06:09.915 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.857+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3ec1a38f0 con 0x7ff3ec100070 2023-07-20T19:06:09.916 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.857+0000 7ff3d3fff700 1 -- 172.21.15.61:0/1823936763 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3e009aa80 con 0x7ff3ec100070 2023-07-20T19:06:09.916 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2023-07-20T19:06:09.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:09 smithi156 ceph-mon[129228]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:09.917 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ff3d40a1150 msgr2=0x7ff3d40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ff3d40a1150 0x7ff3d40a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00a400 tx=0x7ff3dc00d040 comp rx=0 tx=0).stop 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 msgr2=0x7ff3ec19cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 0x7ff3ec19cbf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00095c0 tx=0x7ff3e000f710 comp rx=0 tx=0).stop 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 shutdown_connections 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ff3d40a1150 0x7ff3d40a3610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3ec104580 0x7ff3ec196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.918 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3ec0ff6b0 0x7ff3ec19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.919 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 --2- 172.21.15.61:0/1823936763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3ec100070 0x7ff3ec19cbf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:09.919 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.859+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 >> 172.21.15.61:0/1823936763 conn(0x7ff3ec076770 msgr2=0x7ff3ec103370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:09.919 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.860+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 shutdown_connections 2023-07-20T19:06:09.919 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:09.860+0000 7ff3f1227700 1 -- 172.21.15.61:0/1823936763 wait complete. 2023-07-20T19:06:09.940 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:09 smithi061 ceph-mon[127454]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:10.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:09 smithi149 ceph-mon[128885]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:10.277 INFO:tasks.cephadm:Adding mds.a on smithi061 2023-07-20T19:06:10.277 INFO:tasks.cephadm:Adding mds.d on smithi061 2023-07-20T19:06:10.278 INFO:tasks.cephadm:Adding mds.g on smithi061 2023-07-20T19:06:10.278 INFO:tasks.cephadm:Adding mds.j on smithi061 2023-07-20T19:06:10.278 INFO:tasks.cephadm:Adding mds.b on smithi149 2023-07-20T19:06:10.278 INFO:tasks.cephadm:Adding mds.e on smithi149 2023-07-20T19:06:10.278 INFO:tasks.cephadm:Adding mds.h on smithi149 2023-07-20T19:06:10.279 INFO:tasks.cephadm:Adding mds.k on smithi149 2023-07-20T19:06:10.279 INFO:tasks.cephadm:Adding mds.c on smithi156 2023-07-20T19:06:10.279 INFO:tasks.cephadm:Adding mds.f on smithi156 2023-07-20T19:06:10.279 INFO:tasks.cephadm:Adding mds.i on smithi156 2023-07-20T19:06:10.279 INFO:tasks.cephadm:Adding mds.l on smithi156 2023-07-20T19:06:10.280 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7541eb30-272f-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l' 2023-07-20T19:06:10.853 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T19:06:10.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:10 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1823936763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:10.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:10 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1823936763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:11.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:10 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1823936763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:11.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:11 smithi156 ceph-mon[129228]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:11.969 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:11 smithi061 ceph-mon[127454]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:12.044 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.985+0000 7f81c2430700 1 -- 172.21.15.156:0/4152380300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc1012a0 msgr2=0x7f81bc107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:12.044 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.985+0000 7f81c2430700 1 --2- 172.21.15.156:0/4152380300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc1012a0 0x7f81bc107b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f81b400b0a0 tx=0x7f81b400b3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.045 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.985+0000 7f81c2430700 1 -- 172.21.15.156:0/4152380300 shutdown_connections 2023-07-20T19:06:12.045 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.985+0000 7f81c2430700 1 --2- 172.21.15.156:0/4152380300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc1012a0 0x7f81bc107b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.045 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.985+0000 7f81c2430700 1 --2- 172.21.15.156:0/4152380300 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81bc0ff700 0x7f81bc0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.045 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.985+0000 7f81c2430700 1 --2- 172.21.15.156:0/4152380300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81bc100900 0x7f81bc100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.046 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- 172.21.15.156:0/4152380300 >> 172.21.15.156:0/4152380300 conn(0x7f81bc0fac80 msgr2=0x7f81bc0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:12.046 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- 172.21.15.156:0/4152380300 shutdown_connections 2023-07-20T19:06:12.046 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- 172.21.15.156:0/4152380300 wait complete. 2023-07-20T19:06:12.047 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 Processor -- start 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- start start 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 0x7f81bc195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81bc100900 0x7f81bc1959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81bc1012a0 0x7f81bc19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc19d020 con 0x7f81bc100900 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc19d160 con 0x7f81bc1012a0 2023-07-20T19:06:12.048 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.986+0000 7f81c2430700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc19d2f0 con 0x7f81bc0ff700 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 0x7f81bc195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 0x7f81bc195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48846/0 (socket says 172.21.15.156:48846) 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 -- 172.21.15.156:0/548198552 learned_addr learned my addr 172.21.15.156:0/548198552 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bb7fe700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81bc100900 0x7f81bc1959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 -- 172.21.15.156:0/548198552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81bc1012a0 msgr2=0x7f81bc19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81c09cd700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81bc1012a0 0x7f81bc19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81bc1012a0 0x7f81bc19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.049 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 -- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81bc100900 msgr2=0x7f81bc1959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81bc100900 0x7f81bc1959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 -- 172.21.15.156:0/548198552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b4009d00 con 0x7f81bc0ff700 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81bbfff700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 0x7f81bc195490 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00d9b0 tx=0x7f81ac00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac004020 con 0x7f81bc0ff700 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc19d5d0 con 0x7f81bc0ff700 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81ac009730 con 0x7f81bc0ff700 2023-07-20T19:06:12.050 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac003a40 con 0x7f81bc0ff700 2023-07-20T19:06:12.051 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.987+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc19db20 con 0x7f81bc0ff700 2023-07-20T19:06:12.051 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.988+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc10af10 con 0x7f81bc0ff700 2023-07-20T19:06:12.052 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.989+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f81ac0098a0 con 0x7f81bc0ff700 2023-07-20T19:06:12.052 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.990+0000 7f81b97fa700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f81a40a11d0 0x7f81a40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:12.052 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.990+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81ac0cc540 con 0x7f81bc0ff700 2023-07-20T19:06:12.052 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.990+0000 7f81bb7fe700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f81a40a11d0 0x7f81a40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:12.052 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.991+0000 7f81bb7fe700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f81a40a11d0 0x7f81a40a3690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f81b0005950 tx=0x7f81b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:12.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:11.993+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f81ac096090 con 0x7f81bc0ff700 2023-07-20T19:06:12.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:11 smithi149 ceph-mon[128885]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:12.249 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.190+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f81bc068bf0 con 0x7f81a40a11d0 2023-07-20T19:06:12.258 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.200+0000 7f81b97fa700 1 -- 172.21.15.156:0/548198552 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f81bc068bf0 con 0x7f81a40a11d0 2023-07-20T19:06:12.258 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mds.all update... 2023-07-20T19:06:12.260 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f81a40a11d0 msgr2=0x7f81a40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:12.260 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f81a40a11d0 0x7f81a40a3690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f81b0005950 tx=0x7f81b000b410 comp rx=0 tx=0).stop 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 msgr2=0x7f81bc195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 0x7f81bc195490 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00d9b0 tx=0x7f81ac00dcc0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 shutdown_connections 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f81a40a11d0 0x7f81a40a3690 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81bc1012a0 0x7f81bc19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81bc0ff700 0x7f81bc195490 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 --2- 172.21.15.156:0/548198552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81bc100900 0x7f81bc1959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:12.262 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 >> 172.21.15.156:0/548198552 conn(0x7f81bc0fac80 msgr2=0x7f81bc0fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:12.262 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 shutdown_connections 2023-07-20T19:06:12.262 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-20T19:06:12.202+0000 7f81c2430700 1 -- 172.21.15.156:0/548198552 wait complete. 2023-07-20T19:06:12.599 DEBUG:teuthology.orchestra.run.smithi061:mds.a> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T19:06:12.602 DEBUG:teuthology.orchestra.run.smithi061:mds.d> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T19:06:12.641 INFO:journalctl@ceph.mds.a.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:06:12.643 DEBUG:teuthology.orchestra.run.smithi061:mds.g> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T19:06:12.673 INFO:journalctl@ceph.mds.d.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:06:12.674 DEBUG:teuthology.orchestra.run.smithi061:mds.j> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T19:06:12.703 INFO:journalctl@ceph.mds.g.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:06:12.704 DEBUG:teuthology.orchestra.run.smithi149:mds.b> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T19:06:12.706 DEBUG:teuthology.orchestra.run.smithi149:mds.e> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T19:06:12.731 INFO:journalctl@ceph.mds.j.smithi061.stdout:-- Logs begin at Thu 2023-07-20 18:36:18 UTC. -- 2023-07-20T19:06:12.744 INFO:journalctl@ceph.mds.b.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:06:12.745 DEBUG:teuthology.orchestra.run.smithi149:mds.h> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T19:06:12.774 INFO:journalctl@ceph.mds.e.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:06:12.775 DEBUG:teuthology.orchestra.run.smithi149:mds.k> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T19:06:12.803 INFO:journalctl@ceph.mds.h.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:06:12.803 DEBUG:teuthology.orchestra.run.smithi156:mds.c> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T19:06:12.806 DEBUG:teuthology.orchestra.run.smithi156:mds.f> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T19:06:12.830 INFO:journalctl@ceph.mds.k.smithi149.stdout:-- Logs begin at Thu 2023-07-20 18:36:16 UTC. -- 2023-07-20T19:06:12.834 INFO:journalctl@ceph.mds.c.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:06:12.835 DEBUG:teuthology.orchestra.run.smithi156:mds.i> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T19:06:12.860 INFO:journalctl@ceph.mds.f.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:06:12.861 DEBUG:teuthology.orchestra.run.smithi156:mds.l> sudo journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T19:06:12.887 INFO:journalctl@ceph.mds.i.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:06:12.887 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T19:06:12.888 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T19:06:12.913 INFO:journalctl@ceph.mds.l.smithi156.stdout:-- Logs begin at Thu 2023-07-20 18:36:30 UTC. -- 2023-07-20T19:06:13.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 -- 172.21.15.61:0/480068769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f41011e0 msgr2=0x7fa3f4107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 --2- 172.21.15.61:0/480068769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f41011e0 0x7fa3f4107a90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00b0a0 tx=0x7fa3ec00b3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 -- 172.21.15.61:0/480068769 shutdown_connections 2023-07-20T19:06:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 --2- 172.21.15.61:0/480068769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f41011e0 0x7fa3f4107a90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 --2- 172.21.15.61:0/480068769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f40ff650 0x7fa3f40ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 --2- 172.21.15.61:0/480068769 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3f4100840 0x7fa3f4100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 -- 172.21.15.61:0/480068769 >> 172.21.15.61:0/480068769 conn(0x7fa3f40faba0 msgr2=0x7fa3f40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 -- 172.21.15.61:0/480068769 shutdown_connections 2023-07-20T19:06:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.968+0000 7fa3f882f700 1 -- 172.21.15.61:0/480068769 wait complete. 2023-07-20T19:06:13.029 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 Processor -- start 2023-07-20T19:06:13.029 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 -- start start 2023-07-20T19:06:13.029 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3f40ff650 0x7fa3f41035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.029 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f4100840 0x7fa3f4101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.029 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 0x7fa3f4102160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f4102900 con 0x7fa3f41011e0 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f4102a70 con 0x7fa3f40ff650 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f882f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f41a0530 con 0x7fa3f4100840 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 0x7fa3f4102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f1d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f4100840 0x7fa3f4101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 0x7fa3f4102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58150/0 (socket says 172.21.15.61:58150) 2023-07-20T19:06:13.030 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 -- 172.21.15.61:0/3311286966 learned_addr learned my addr 172.21.15.61:0/3311286966 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f259c700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3f40ff650 0x7fa3f41035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 -- 172.21.15.61:0/3311286966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f4100840 msgr2=0x7fa3f4101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f4100840 0x7fa3f4101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 -- 172.21.15.61:0/3311286966 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3f40ff650 msgr2=0x7fa3f41035e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3f40ff650 0x7fa3f41035e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 -- 172.21.15.61:0/3311286966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec009d00 con 0x7fa3f41011e0 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.969+0000 7fa3f2d9d700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 0x7fa3f4102160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00b780 tx=0x7fa3ec00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.970+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00e040 con 0x7fa3f41011e0 2023-07-20T19:06:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.970+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f41a07b0 con 0x7fa3f41011e0 2023-07-20T19:06:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.970+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3ec004e60 con 0x7fa3f41011e0 2023-07-20T19:06:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.970+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0053b0 con 0x7fa3f41011e0 2023-07-20T19:06:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.970+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f41a0c30 con 0x7fa3f41011e0 2023-07-20T19:06:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.971+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f410ae70 con 0x7fa3f41011e0 2023-07-20T19:06:13.033 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.972+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa3ec01d470 con 0x7fa3f41011e0 2023-07-20T19:06:13.033 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.972+0000 7fa3eb7fe700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa3e00a11a0 0x7fa3e00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.033 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.972+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa3ec0cdf20 con 0x7fa3f41011e0 2023-07-20T19:06:13.033 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.972+0000 7fa3f259c700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa3e00a11a0 0x7fa3e00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.034 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.973+0000 7fa3f259c700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa3e00a11a0 0x7fa3e00a3660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3f41006a0 tx=0x7fa3e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:13.034 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:12.976+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa3ec097ba0 con 0x7fa3f41011e0 2023-07-20T19:06:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.176+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3f410b030 con 0x7fa3f41011e0 2023-07-20T19:06:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.177+0000 7fa3eb7fe700 1 -- 172.21.15.61:0/3311286966 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18408 (secure 0 0 0) 0x7fa3ec09b1c0 con 0x7fa3f41011e0 2023-07-20T19:06:13.236 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:13.236 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","created":"2023-07-20T18:59:47.496551+0000","modified":"2023-07-20T19:06:04.764327+0000","last_up_change":"2023-07-20T19:06:03.755886+0000","last_in_change":"2023-07-20T19:05:46.242910+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T19:02:56.142587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfae06af-18f4-487e-ae4b-8e8c9d36ef76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6803","nonce":1072462099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6805","nonce":1072462099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6809","nonce":1072462099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6807","nonce":1072462099}]},"public_addr":"172.21.15.61:6803/1072462099","cluster_addr":"172.21.15.61:6805/1072462099","heartbeat_back_addr":"172.21.15.61:6809/1072462099","heartbeat_front_addr":"172.21.15.61:6807/1072462099","state":["exists","up"]},{"osd":1,"uuid":"d3e2b1d4-abe5-4438-9196-6ef26230a071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6801","nonce":414280084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6803","nonce":414280084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6807","nonce":414280084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6805","nonce":414280084}]},"public_addr":"172.21.15.149:6801/414280084","cluster_addr":"172.21.15.149:6803/414280084","heartbeat_back_addr":"172.21.15.149:6807/414280084","heartbeat_front_addr":"172.21.15.149:6805/414280084","state":["exists","up"]},{"osd":2,"uuid":"34a03029-87a9-49e1-b262-32be5070df0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6801","nonce":738107812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6803","nonce":738107812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6807","nonce":738107812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6805","nonce":738107812}]},"public_addr":"172.21.15.156:6801/738107812","cluster_addr":"172.21.15.156:6803/738107812","heartbeat_back_addr":"172.21.15.156:6807/738107812","heartbeat_front_addr":"172.21.15.156:6805/738107812","state":["exists","up"]},{"osd":3,"uuid":"c5711133-530f-488a-9db4-2b8a61b7ea02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6811","nonce":1371683823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6813","nonce":1371683823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6817","nonce":1371683823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6815","nonce":1371683823}]},"public_addr":"172.21.15.61:6811/1371683823","cluster_addr":"172.21.15.61:6813/1371683823","heartbeat_back_addr":"172.21.15.61:6817/1371683823","heartbeat_front_addr":"172.21.15.61:6815/1371683823","state":["exists","up"]},{"osd":4,"uuid":"6062cdd3-5bf1-458a-8454-253627db736f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6809","nonce":3513915816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6811","nonce":3513915816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6815","nonce":3513915816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6813","nonce":3513915816}]},"public_addr":"172.21.15.149:6809/3513915816","cluster_addr":"172.21.15.149:6811/3513915816","heartbeat_back_addr":"172.21.15.149:6815/3513915816","heartbeat_front_addr":"172.21.15.149:6813/3513915816","state":["exists","up"]},{"osd":5,"uuid":"1fae1b15-43cb-40d9-ad90-73af4cc96197","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6809","nonce":211953844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6811","nonce":211953844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6815","nonce":211953844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6813","nonce":211953844}]},"public_addr":"172.21.15.156:6809/211953844","cluster_addr":"172.21.15.156:6811/211953844","heartbeat_back_addr":"172.21.15.156:6815/211953844","heartbeat_front_addr":"172.21.15.156:6813/211953844","state":["exists","up"]},{"osd":6,"uuid":"84316d15-3a51-432c-9d02-57270ac09769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6819","nonce":1770665297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6821","nonce":1770665297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6825","nonce":1770665297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6823","nonce":1770665297}]},"public_addr":"172.21.15.61:6819/1770665297","cluster_addr":"172.21.15.61:6821/1770665297","heartbeat_back_addr":"172.21.15.61:6825/1770665297","heartbeat_front_addr":"172.21.15.61:6823/1770665297","state":["exists","up"]},{"osd":7,"uuid":"705c1527-9342-402c-add4-4382e005dbfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6817","nonce":1095358817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6819","nonce":1095358817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6823","nonce":1095358817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6821","nonce":1095358817}]},"public_addr":"172.21.15.149:6817/1095358817","cluster_addr":"172.21.15.149:6819/1095358817","heartbeat_back_addr":"172.21.15.149:6823/1095358817","heartbeat_front_addr":"172.21.15.149:6821/1095358817","state":["exists","up"]},{"osd":8,"uuid":"6e790866-ee13-4c27-bbd4-ea90e853de1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6817","nonce":1602221626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6819","nonce":1602221626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6823","nonce":1602221626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6821","nonce":1602221626}]},"public_addr":"172.21.15.156:6817/1602221626","cluster_addr":"172.21.15.156:6819/1602221626","heartbeat_back_addr":"172.21.15.156:6823/1602221626","heartbeat_front_addr":"172.21.15.156:6821/1602221626","state":["exists","up"]},{"osd":9,"uuid":"ac0b69d5-ccea-470d-a8c8-b9a2f508af24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6827","nonce":2654664682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6829","nonce":2654664682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6833","nonce":2654664682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6831","nonce":2654664682}]},"public_addr":"172.21.15.61:6827/2654664682","cluster_addr":"172.21.15.61:6829/2654664682","heartbeat_back_addr":"172.21.15.61:6833/2654664682","heartbeat_front_addr":"172.21.15.61:6831/2654664682","state":["exists","up"]},{"osd":10,"uuid":"1a6e7d98-a3f2-4d81-843e-bca056ec748c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6825","nonce":1912645375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6827","nonce":1912645375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6831","nonce":1912645375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6829","nonce":1912645375}]},"public_addr":"172.21.15.149:6825/1912645375","cluster_addr":"172.21.15.149:6827/1912645375","heartbeat_back_addr":"172.21.15.149:6831/1912645375","heartbeat_front_addr":"172.21.15.149:6829/1912645375","state":["exists","up"]},{"osd":11,"uuid":"5bcb26b2-0789-4cf8-9803-98a6da33d5d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6825","nonce":893354887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6827","nonce":893354887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6831","nonce":893354887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6829","nonce":893354887}]},"public_addr":"172.21.15.156:6825/893354887","cluster_addr":"172.21.15.156:6827/893354887","heartbeat_back_addr":"172.21.15.156:6831/893354887","heartbeat_front_addr":"172.21.15.156:6829/893354887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:09.841610+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:31.306806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:52.704303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:13.230276+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:34.832690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:55.534516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:16.097468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:37.618136+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:58.685044+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:19.824839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:41.039703+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:06:02.038214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.61:6801/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6800/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/3344679989":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6801/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/170145058":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/575168883":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/2333936767":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/74416212":"2023-07-21T19:00:21.461577+0000","172.21.15.61:6800/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/2426181232":"2023-07-21T19:00:21.461577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T19:06:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa3e00a11a0 msgr2=0x7fa3e00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa3e00a11a0 0x7fa3e00a3660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3f41006a0 tx=0x7fa3e4009260 comp rx=0 tx=0).stop 2023-07-20T19:06:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 msgr2=0x7fa3f4102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 0x7fa3f4102160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00b780 tx=0x7fa3ec00bd60 comp rx=0 tx=0).stop 2023-07-20T19:06:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 shutdown_connections 2023-07-20T19:06:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa3e00a11a0 0x7fa3e00a3660 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.240 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3f41011e0 0x7fa3f4102160 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.240 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3f40ff650 0x7fa3f41035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.240 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 --2- 172.21.15.61:0/3311286966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3f4100840 0x7fa3f4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.241 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.179+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 >> 172.21.15.61:0/3311286966 conn(0x7fa3f40faba0 msgr2=0x7fa3f40679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:13.241 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.180+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 shutdown_connections 2023-07-20T19:06:13.241 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.180+0000 7fa3f882f700 1 -- 172.21.15.61:0/3311286966 wait complete. 2023-07-20T19:06:13.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T19:02:56.142587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T19:06:13.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T19:06:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 -- 172.21.15.61:0/3736443229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 msgr2=0x7f05700ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 --2- 172.21.15.61:0/3736443229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700ffa80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f056c0099c0 tx=0x7f056c009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 -- 172.21.15.61:0/3736443229 shutdown_connections 2023-07-20T19:06:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 --2- 172.21.15.61:0/3736443229 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f05701011f0 0x7f0570107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 --2- 172.21.15.61:0/3736443229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700ffa80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 --2- 172.21.15.61:0/3736443229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0570100850 0x7f0570100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.322+0000 7f0578018700 1 -- 172.21.15.61:0/3736443229 >> 172.21.15.61:0/3736443229 conn(0x7f05700fabd0 msgr2=0x7f05700fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 -- 172.21.15.61:0/3736443229 shutdown_connections 2023-07-20T19:06:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 -- 172.21.15.61:0/3736443229 wait complete. 2023-07-20T19:06:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 Processor -- start 2023-07-20T19:06:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 -- start start 2023-07-20T19:06:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700684a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0570100850 0x7f057019b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f05701011f0 0x7f057019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057019dff0 con 0x7f05700ff660 2023-07-20T19:06:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057019e160 con 0x7f05701011f0 2023-07-20T19:06:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0578018700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057019e320 con 0x7f0570100850 2023-07-20T19:06:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0575db4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700684a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.323+0000 7f0575db4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700684a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58160/0 (socket says 172.21.15.61:58160) 2023-07-20T19:06:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 -- 172.21.15.61:0/1385512122 learned_addr learned my addr 172.21.15.61:0/1385512122 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 -- 172.21.15.61:0/1385512122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0570100850 msgr2=0x7f057019b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0570100850 0x7f057019b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 -- 172.21.15.61:0/1385512122 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f05701011f0 msgr2=0x7f057019d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f05765b5700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f05701011f0 0x7f057019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f05701011f0 0x7f057019d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 -- 172.21.15.61:0/1385512122 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0095f0 con 0x7f05700ff660 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0575db4700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700684a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f056c009fd0 tx=0x7f056c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c01d070 con 0x7f05700ff660 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f056c0054f0 con 0x7f05700ff660 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057019e5a0 con 0x7f05700ff660 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c004c40 con 0x7f05700ff660 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.324+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057019e9c0 con 0x7f05700ff660 2023-07-20T19:06:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.325+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057010ae80 con 0x7f05700ff660 2023-07-20T19:06:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.326+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f056c00fc30 con 0x7f05700ff660 2023-07-20T19:06:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.327+0000 7f0566ffd700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f055c0a1150 0x7f055c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.327+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f056c0cd7a0 con 0x7f05700ff660 2023-07-20T19:06:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.327+0000 7f05755b3700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f055c0a1150 0x7f055c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.327+0000 7f05755b3700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f055c0a1150 0x7f055c0a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0560005950 tx=0x7f05600058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.330+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f056c097420 con 0x7f05700ff660 2023-07-20T19:06:13.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:13.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:06:13.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: Saving service mds.all spec with placement smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-20T19:06:13.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:13.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:13.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:13.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:13.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:13.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:13.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:13.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:13.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: Deploying daemon mds.l on smithi156 2023-07-20T19:06:13.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:13 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3311286966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:13.561 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:13.561 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:06:13.562 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: Saving service mds.all spec with placement smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-20T19:06:13.562 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:13.562 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:13.563 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:13.563 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:13.563 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:13.563 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:13.564 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:13.564 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:13.564 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: Deploying daemon mds.l on smithi156 2023-07-20T19:06:13.565 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:13 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3311286966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:13.587 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.529+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0570106620 con 0x7f05700ff660 2023-07-20T19:06:13.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.530+0000 7f0566ffd700 1 -- 172.21.15.61:0/1385512122 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f056c09aa40 con 0x7f05700ff660 2023-07-20T19:06:13.588 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2023-07-20T19:06:13.590 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f055c0a1150 msgr2=0x7f055c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.590 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f055c0a1150 0x7f055c0a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0560005950 tx=0x7f05600058e0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 msgr2=0x7f05700684a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700684a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f056c009fd0 tx=0x7f056c00f740 comp rx=0 tx=0).stop 2023-07-20T19:06:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 shutdown_connections 2023-07-20T19:06:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f055c0a1150 0x7f055c0a3610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f05701011f0 0x7f057019d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05700ff660 0x7f05700684a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 --2- 172.21.15.61:0/1385512122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0570100850 0x7f057019b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 >> 172.21.15.61:0/1385512122 conn(0x7f05700fabd0 msgr2=0x7f05700fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 shutdown_connections 2023-07-20T19:06:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.532+0000 7f0578018700 1 -- 172.21.15.61:0/1385512122 wait complete. 2023-07-20T19:06:13.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:13.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:06:13.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: Saving service mds.all spec with placement smithi061=a;smithi061=d;smithi061=g;smithi061=j;smithi149=b;smithi149=e;smithi149=h;smithi149=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-20T19:06:13.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:13.602 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:13.603 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: Deploying daemon mds.l on smithi156 2023-07-20T19:06:13.604 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:13 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3311286966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:13.605 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 -- 172.21.15.61:0/3247607051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 msgr2=0x7f1a30107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3247607051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30107b60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800b0a0 tx=0x7f1a2800b3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 -- 172.21.15.61:0/3247607051 shutdown_connections 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3247607051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30107b60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3247607051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a30100c60 0x7f1a30101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3247607051 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a300ff900 0x7f1a300ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 -- 172.21.15.61:0/3247607051 >> 172.21.15.61:0/3247607051 conn(0x7f1a300fa330 msgr2=0x7f1a300fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 -- 172.21.15.61:0/3247607051 shutdown_connections 2023-07-20T19:06:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.672+0000 7f1a370e1700 1 -- 172.21.15.61:0/3247607051 wait complete. 2023-07-20T19:06:13.733 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 Processor -- start 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 -- start start 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a300ff900 0x7f1a301964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a30100c60 0x7f1a3019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3019e070 con 0x7f1a301002c0 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3019e1b0 con 0x7f1a30100c60 2023-07-20T19:06:13.734 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a370e1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3019e340 con 0x7f1a300ff900 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58176/0 (socket says 172.21.15.61:58176) 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a34e7d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a300ff900 0x7f1a301964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 -- 172.21.15.61:0/3487695624 learned_addr learned my addr 172.21.15.61:0/3487695624 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a3567e700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a30100c60 0x7f1a3019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 -- 172.21.15.61:0/3487695624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a300ff900 msgr2=0x7f1a301964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.735 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a300ff900 0x7f1a301964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 -- 172.21.15.61:0/3487695624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a30100c60 msgr2=0x7f1a3019daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a30100c60 0x7f1a3019daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 -- 172.21.15.61:0/3487695624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28009d00 con 0x7f1a301002c0 2023-07-20T19:06:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a34e7d700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a300ff900 0x7f1a301964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:06:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2ffff700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30196a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00d980 tx=0x7f1a1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.673+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c004020 con 0x7f1a301002c0 2023-07-20T19:06:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.674+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a1c009730 con 0x7f1a301002c0 2023-07-20T19:06:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.674+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3019e620 con 0x7f1a301002c0 2023-07-20T19:06:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.674+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c003a80 con 0x7f1a301002c0 2023-07-20T19:06:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.674+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3019eb30 con 0x7f1a301002c0 2023-07-20T19:06:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.675+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3010ae80 con 0x7f1a301002c0 2023-07-20T19:06:13.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.676+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1a1c0098a0 con 0x7f1a301002c0 2023-07-20T19:06:13.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.676+0000 7f1a2dffb700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1a200a11a0 0x7f1a200a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:13.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.677+0000 7f1a34e7d700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1a200a11a0 0x7f1a200a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:13.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.677+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1a1c0cc140 con 0x7f1a301002c0 2023-07-20T19:06:13.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.677+0000 7f1a34e7d700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1a200a11a0 0x7f1a200a3660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1a30067670 tx=0x7f1a24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:13.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.681+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1a1c095d30 con 0x7f1a301002c0 2023-07-20T19:06:13.986 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.928+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a3010b0a0 con 0x7f1a301002c0 2023-07-20T19:06:13.987 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.929+0000 7f1a2dffb700 1 -- 172.21.15.61:0/3487695624 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f1a1c099350 con 0x7f1a301002c0 2023-07-20T19:06:13.987 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:13.987 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T19:06:13.989 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1a200a11a0 msgr2=0x7f1a200a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.989 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1a200a11a0 0x7f1a200a3660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1a30067670 tx=0x7f1a24009260 comp rx=0 tx=0).stop 2023-07-20T19:06:13.989 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 msgr2=0x7f1a30196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30196a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00d980 tx=0x7f1a1c00dc90 comp rx=0 tx=0).stop 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 shutdown_connections 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1a200a11a0 0x7f1a200a3660 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a30100c60 0x7f1a3019daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a300ff900 0x7f1a301964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 --2- 172.21.15.61:0/3487695624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a301002c0 0x7f1a30196a30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 >> 172.21.15.61:0/3487695624 conn(0x7f1a300fa330 msgr2=0x7f1a300fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:13.990 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 shutdown_connections 2023-07-20T19:06:13.991 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:13.931+0000 7f1a370e1700 1 -- 172.21.15.61:0/3487695624 wait complete. 2023-07-20T19:06:13.991 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 1 2023-07-20T19:06:14.167 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:06:13 smithi156 systemd[1]: Starting Ceph mds.l for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:14.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:14 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1385512122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:14.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:14 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3487695624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:14.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:14 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1385512122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:14.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:14 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3487695624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:14.669 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:06:14 smithi156 podman[154470]: 2023-07-20T19:06:14.669 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:06:14 smithi156 bash[154470]: 982f0730243224fa9f09802ef70c885a741fa4891afb30be9a446dfeffd9264e 2023-07-20T19:06:14.669 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:06:14 smithi156 ceph-mds[154511]: -- 172.21.15.156:0/458672754 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f004cd1e0 con 0x559f004a3000 2023-07-20T19:06:14.669 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:06:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-l[154486]: starting mds.l at 2023-07-20T19:06:14.669 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:06:14 smithi156 systemd[1]: Started Ceph mds.l for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:14.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:14 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1385512122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:14.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:14 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3487695624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:15.793 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:15.793 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6832/2641203790,v1:172.21.15.156:6833/2641203790] up:boot 2023-07-20T19:06:15.793 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: fsmap 1 up:standby 2023-07-20T19:06:15.793 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T19:06:15.793 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.794 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.794 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.794 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:15.794 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:15.794 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:15.794 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:15 smithi149 ceph-mon[128885]: Deploying daemon mds.k on smithi149 2023-07-20T19:06:15.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:15.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6832/2641203790,v1:172.21.15.156:6833/2641203790] up:boot 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: fsmap 1 up:standby 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:15.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:15.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:15.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:15 smithi156 ceph-mon[129228]: Deploying daemon mds.k on smithi149 2023-07-20T19:06:15.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:15.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6832/2641203790,v1:172.21.15.156:6833/2641203790] up:boot 2023-07-20T19:06:15.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: fsmap 1 up:standby 2023-07-20T19:06:15.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T19:06:15.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:15.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:15.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:15.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:15.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:15 smithi061 ceph-mon[127454]: Deploying daemon mds.k on smithi149 2023-07-20T19:06:16.003 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 -- 172.21.15.61:0/2889946700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c100850 msgr2=0x7ffa9c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 --2- 172.21.15.61:0/2889946700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c100850 0x7ffa9c100cb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa980099c0 tx=0x7ffa98009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 -- 172.21.15.61:0/2889946700 shutdown_connections 2023-07-20T19:06:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 --2- 172.21.15.61:0/2889946700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa9c1011f0 0x7ffa9c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 --2- 172.21.15.61:0/2889946700 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 --2- 172.21.15.61:0/2889946700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c100850 0x7ffa9c100cb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 -- 172.21.15.61:0/2889946700 >> 172.21.15.61:0/2889946700 conn(0x7ffa9c0fab90 msgr2=0x7ffa9c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 -- 172.21.15.61:0/2889946700 shutdown_connections 2023-07-20T19:06:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.084+0000 7ffaa373e700 1 -- 172.21.15.61:0/2889946700 wait complete. 2023-07-20T19:06:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 Processor -- start 2023-07-20T19:06:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 -- start start 2023-07-20T19:06:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c195250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa9c100850 0x7ffa9c195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa9c1011f0 0x7ffa9c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:16.148 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19cdd0 con 0x7ffa9c0ff660 2023-07-20T19:06:16.148 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19cf10 con 0x7ffa9c1011f0 2023-07-20T19:06:16.148 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa373e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19d0a0 con 0x7ffa9c100850 2023-07-20T19:06:16.148 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa14da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c195250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa14da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c195250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58188/0 (socket says 172.21.15.61:58188) 2023-07-20T19:06:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa14da700 1 -- 172.21.15.61:0/1865653034 learned_addr learned my addr 172.21.15.61:0/1865653034 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.085+0000 7ffaa1cdb700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa9c1011f0 0x7ffa9c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa0cd9700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa9c100850 0x7ffa9c195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa14da700 1 -- 172.21.15.61:0/1865653034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa9c100850 msgr2=0x7ffa9c195790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa14da700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa9c100850 0x7ffa9c195790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa14da700 1 -- 172.21.15.61:0/1865653034 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa9c1011f0 msgr2=0x7ffa9c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa14da700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa9c1011f0 0x7ffa9c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa14da700 1 -- 172.21.15.61:0/1865653034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa980095f0 con 0x7ffa9c0ff660 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa14da700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c195250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c990 tx=0x7ffa9000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90007b00 con 0x7ffa9c0ff660 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa90007c60 con 0x7ffa9c0ff660 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c19d380 con 0x7ffa9c0ff660 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90018660 con 0x7ffa9c0ff660 2023-07-20T19:06:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.086+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c19d890 con 0x7ffa9c0ff660 2023-07-20T19:06:16.152 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.087+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c106b50 con 0x7ffa9c0ff660 2023-07-20T19:06:16.152 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.088+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffa9000f450 con 0x7ffa9c0ff660 2023-07-20T19:06:16.153 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.089+0000 7ffa8e7fc700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa880a11a0 0x7ffa880a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:16.153 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.089+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffa900cc500 con 0x7ffa9c0ff660 2023-07-20T19:06:16.153 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.089+0000 7ffaa0cd9700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa880a11a0 0x7ffa880a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:16.153 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.089+0000 7ffaa0cd9700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa880a11a0 0x7ffa880a3660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffa98000c00 tx=0x7ffa98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:16.153 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.092+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffa900960e0 con 0x7ffa9c0ff660 2023-07-20T19:06:16.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.338+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa9c02cbe0 con 0x7ffa9c0ff660 2023-07-20T19:06:16.397 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.338+0000 7ffa8e7fc700 1 -- 172.21.15.61:0/1865653034 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7ffa90099700 con 0x7ffa9c0ff660 2023-07-20T19:06:16.398 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:16.398 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T19:06:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa880a11a0 msgr2=0x7ffa880a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa880a11a0 0x7ffa880a3660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffa98000c00 tx=0x7ffa98005fb0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 msgr2=0x7ffa9c195250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c195250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c990 tx=0x7ffa9000cca0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 shutdown_connections 2023-07-20T19:06:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7ffa880a11a0 0x7ffa880a3660 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa9c1011f0 0x7ffa9c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa9c0ff660 0x7ffa9c195250 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 --2- 172.21.15.61:0/1865653034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa9c100850 0x7ffa9c195790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 >> 172.21.15.61:0/1865653034 conn(0x7ffa9c0fab90 msgr2=0x7ffa9c103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 shutdown_connections 2023-07-20T19:06:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:16.340+0000 7ffaa373e700 1 -- 172.21.15.61:0/1865653034 wait complete. 2023-07-20T19:06:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 2 2023-07-20T19:06:16.407 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:06:16 smithi149 systemd[1]: Starting Ceph mds.k for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:16.680 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:06:16 smithi149 podman[152908]: 2023-07-20T19:06:16.681 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:16 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1865653034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:16.719 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:16 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1865653034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:16.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:16 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1865653034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:17.054 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:06:16 smithi149 ceph-mds[152948]: -- 172.21.15.149:0/1475486765 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b1d85471e0 con 0x55b1d851cc00 2023-07-20T19:06:17.055 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:06:16 smithi149 bash[152908]: b4e5fe137ae04a39dc2a9fda337bd165d355e9277a1a1bc3ab5484b1ac50fbed 2023-07-20T19:06:17.055 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:06:16 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-k[152924]: starting mds.k at 2023-07-20T19:06:17.055 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:06:16 smithi149 systemd[1]: Started Ceph mds.k for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:17.658 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:17.658 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.658 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.658 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.658 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:17.658 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:17.659 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:17 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:17.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:17.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:17.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:17.806 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:17 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:17.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:17.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:17.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:17.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:17.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:17 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:18.184 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:06:18 smithi061 systemd[1]: Starting Ceph mds.j for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:18.412 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:18.555 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 -- 172.21.15.61:0/607100871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 msgr2=0x7fc060107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:18.555 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 --2- 172.21.15.61:0/607100871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc060107a90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc05800b0d0 tx=0x7fc05800b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.556 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 -- 172.21.15.61:0/607100871 shutdown_connections 2023-07-20T19:06:18.556 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 --2- 172.21.15.61:0/607100871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc060107a90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.556 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 --2- 172.21.15.61:0/607100871 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc0600ff650 0x7fc0600ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.557 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 --2- 172.21.15.61:0/607100871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc060100840 0x7fc060100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.557 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 -- 172.21.15.61:0/607100871 >> 172.21.15.61:0/607100871 conn(0x7fc0600faba0 msgr2=0x7fc0600fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:18.557 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.492+0000 7fc065ce6700 1 -- 172.21.15.61:0/607100871 shutdown_connections 2023-07-20T19:06:18.557 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.493+0000 7fc065ce6700 1 -- 172.21.15.61:0/607100871 wait complete. 2023-07-20T19:06:18.558 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.493+0000 7fc065ce6700 1 Processor -- start 2023-07-20T19:06:18.558 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.493+0000 7fc065ce6700 1 -- start start 2023-07-20T19:06:18.558 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc065ce6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc0600ff650 0x7fc060195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc065ce6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc060100840 0x7fc0601959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc065ce6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc06019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc065ce6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06019cfe0 con 0x7fc0601011e0 2023-07-20T19:06:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc065ce6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06019d120 con 0x7fc0600ff650 2023-07-20T19:06:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc065ce6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06019d2b0 con 0x7fc060100840 2023-07-20T19:06:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc06019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc0600ff650 0x7fc060195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc06019ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58204/0 (socket says 172.21.15.61:58204) 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 -- 172.21.15.61:0/3609462789 learned_addr learned my addr 172.21.15.61:0/3609462789 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05effd700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc060100840 0x7fc0601959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 -- 172.21.15.61:0/3609462789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc060100840 msgr2=0x7fc0601959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc060100840 0x7fc0601959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 -- 172.21.15.61:0/3609462789 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc0600ff650 msgr2=0x7fc060195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc0600ff650 0x7fc060195460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 -- 172.21.15.61:0/3609462789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc058009d00 con 0x7fc0601011e0 2023-07-20T19:06:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05ffff700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc06019ca10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc058007a00 tx=0x7fc0580094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.494+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05800e040 con 0x7fc0601011e0 2023-07-20T19:06:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.495+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06019d530 con 0x7fc0601011e0 2023-07-20T19:06:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.495+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc058003e10 con 0x7fc0601011e0 2023-07-20T19:06:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.495+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058012b10 con 0x7fc0601011e0 2023-07-20T19:06:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.495+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06019d9e0 con 0x7fc0601011e0 2023-07-20T19:06:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.496+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06010af10 con 0x7fc0601011e0 2023-07-20T19:06:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.497+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc058012cd0 con 0x7fc0601011e0 2023-07-20T19:06:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.497+0000 7fc05cff9700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fc0480a11a0 0x7fc0480a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.498+0000 7fc05f7fe700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fc0480a11a0 0x7fc0480a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.498+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc0580cdc40 con 0x7fc0601011e0 2023-07-20T19:06:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.498+0000 7fc05f7fe700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fc0480a11a0 0x7fc0480a3660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0601006a0 tx=0x7fc054009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.502+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc0580978c0 con 0x7fc0601011e0 2023-07-20T19:06:18.720 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:06:18 smithi061 podman[160511]: 2023-07-20T19:06:18.720 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:06:18 smithi061 bash[160511]: aa07e606e87924280d8402b81bc13a0b99eb4888af752d9ba749d6c2ee253818 2023-07-20T19:06:18.720 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:06:18 smithi061 systemd[1]: Started Ceph mds.j for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:18.720 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:06:18 smithi061 ceph-mds[160595]: -- 172.21.15.61:0/4241416383 <== mon.0 v2:172.21.15.61:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f43122d1e0 con 0x55f431202c00 2023-07-20T19:06:18.720 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:06:18 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-j[160550]: starting mds.j at 2023-07-20T19:06:18.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:18 smithi061 ceph-mon[127454]: Deploying daemon mds.j on smithi061 2023-07-20T19:06:18.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:18 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.149:6832/1646752575,v1:172.21.15.149:6833/1646752575] up:boot 2023-07-20T19:06:18.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:18 smithi061 ceph-mon[127454]: fsmap 2 up:standby 2023-07-20T19:06:18.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:18 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T19:06:18.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:18 smithi149 ceph-mon[128885]: Deploying daemon mds.j on smithi061 2023-07-20T19:06:18.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:18 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.149:6832/1646752575,v1:172.21.15.149:6833/1646752575] up:boot 2023-07-20T19:06:18.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:18 smithi149 ceph-mon[128885]: fsmap 2 up:standby 2023-07-20T19:06:18.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:18 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T19:06:18.818 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.759+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0601064a0 con 0x7fc0601011e0 2023-07-20T19:06:18.819 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.760+0000 7fc05cff9700 1 -- 172.21.15.61:0/3609462789 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7fc0580056c0 con 0x7fc0601011e0 2023-07-20T19:06:18.819 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:18.819 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T19:06:18.821 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fc0480a11a0 msgr2=0x7fc0480a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:18.821 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fc0480a11a0 0x7fc0480a3660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0601006a0 tx=0x7fc054009190 comp rx=0 tx=0).stop 2023-07-20T19:06:18.821 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 msgr2=0x7fc06019ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:18.821 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc06019ca10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc058007a00 tx=0x7fc0580094c0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 shutdown_connections 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fc0480a11a0 0x7fc0480a3660 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0601011e0 0x7fc06019ca10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc0600ff650 0x7fc060195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 --2- 172.21.15.61:0/3609462789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc060100840 0x7fc0601959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 >> 172.21.15.61:0/3609462789 conn(0x7fc0600faba0 msgr2=0x7fc0600fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 shutdown_connections 2023-07-20T19:06:18.822 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:18.762+0000 7fc065ce6700 1 -- 172.21.15.61:0/3609462789 wait complete. 2023-07-20T19:06:18.823 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 3 2023-07-20T19:06:18.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:18 smithi156 ceph-mon[129228]: Deploying daemon mds.j on smithi061 2023-07-20T19:06:18.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:18 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.149:6832/1646752575,v1:172.21.15.149:6833/1646752575] up:boot 2023-07-20T19:06:18.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:18 smithi156 ceph-mon[129228]: fsmap 2 up:standby 2023-07-20T19:06:18.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:18 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T19:06:19.932 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:19.932 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:19.932 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:19.932 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3609462789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.61:6834/1326256186,v1:172.21.15.61:6835/1326256186] up:boot 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: fsmap 3 up:standby 2023-07-20T19:06:19.933 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:19 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T19:06:20.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:20.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:20.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3609462789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.61:6834/1326256186,v1:172.21.15.61:6835/1326256186] up:boot 2023-07-20T19:06:20.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: fsmap 3 up:standby 2023-07-20T19:06:20.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:19 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T19:06:20.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:20.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:20.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:20.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:20.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:20.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:20.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:20.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3609462789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:20.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.61:6834/1326256186,v1:172.21.15.61:6835/1326256186] up:boot 2023-07-20T19:06:20.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: fsmap 3 up:standby 2023-07-20T19:06:20.169 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:19 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T19:06:20.182 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:06:20 smithi061 systemd[1]: Starting Ceph mds.d for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:20.715 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:06:20 smithi061 podman[160900]: 2023-07-20T19:06:20.835 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:20.968 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 -- 172.21.15.61:0/1642643903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa238100840 msgr2=0x7fa238100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:20.968 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 --2- 172.21.15.61:0/1642643903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa238100840 0x7fa238100ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa2200099c0 tx=0x7fa220009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:20.968 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 -- 172.21.15.61:0/1642643903 shutdown_connections 2023-07-20T19:06:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 --2- 172.21.15.61:0/1642643903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2381011e0 0x7fa238107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 --2- 172.21.15.61:0/1642643903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2380ff650 0x7fa2380ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 --2- 172.21.15.61:0/1642643903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa238100840 0x7fa238100ca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 -- 172.21.15.61:0/1642643903 >> 172.21.15.61:0/1642643903 conn(0x7fa2380faba0 msgr2=0x7fa2380fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 -- 172.21.15.61:0/1642643903 shutdown_connections 2023-07-20T19:06:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.908+0000 7fa23df03700 1 -- 172.21.15.61:0/1642643903 wait complete. 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.909+0000 7fa23df03700 1 Processor -- start 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.909+0000 7fa23df03700 1 -- start start 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 0x7fa2381953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa238100840 0x7fa2381958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2381011e0 0x7fa23819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23819cf20 con 0x7fa2380ff650 2023-07-20T19:06:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23819d060 con 0x7fa238100840 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23819d1f0 con 0x7fa2381011e0 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 0x7fa2381953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 0x7fa2381953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58240/0 (socket says 172.21.15.61:58240) 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 -- 172.21.15.61:0/3853685059 learned_addr learned my addr 172.21.15.61:0/3853685059 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa236ffd700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa238100840 0x7fa2381958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 -- 172.21.15.61:0/3853685059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2381011e0 msgr2=0x7fa23819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:20.972 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2381011e0 0x7fa23819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:20.973 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 -- 172.21.15.61:0/3853685059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa238100840 msgr2=0x7fa2381958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:20.973 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa238100840 0x7fa2381958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:20.973 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 -- 172.21.15.61:0/3853685059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200095f0 con 0x7fa2380ff650 2023-07-20T19:06:20.973 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa2377fe700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 0x7fa2381953a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa22800beb0 tx=0x7fa22800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:20.973 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:06:20 smithi061 ceph-mds[160940]: -- 172.21.15.61:0/2393049205 <== mon.0 v2:172.21.15.61:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561610e731e0 con 0x561610e48800 2023-07-20T19:06:20.974 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:06:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-d[160916]: starting mds.d at 2023-07-20T19:06:20.974 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:06:20 smithi061 bash[160900]: af58bae48cb66b87d986dd544ee0b740f5c37bbcb4c273e71ad571bd0b9d23ef 2023-07-20T19:06:20.974 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:06:20 smithi061 systemd[1]: Started Ceph mds.d for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:20.974 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:20 smithi061 ceph-mon[127454]: Deploying daemon mds.d on smithi061 2023-07-20T19:06:20.975 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228004220 con 0x7fa2380ff650 2023-07-20T19:06:20.975 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa228004380 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa238067930 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.910+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228003c70 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.911+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa238067e90 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.911+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23810af30 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.913+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa228009730 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.913+0000 7fa234ff9700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1220 0x7fa2240a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.913+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2280cdee0 con 0x7fa2380ff650 2023-07-20T19:06:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.913+0000 7fa236ffd700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1220 0x7fa2240a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:20.977 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.914+0000 7fa236ffd700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1220 0x7fa2240a36e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa220000c00 tx=0x7fa220009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:20.977 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:20.917+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa228097ad0 con 0x7fa2380ff650 2023-07-20T19:06:21.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:20 smithi149 ceph-mon[128885]: Deploying daemon mds.d on smithi061 2023-07-20T19:06:21.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:20 smithi156 ceph-mon[129228]: Deploying daemon mds.d on smithi061 2023-07-20T19:06:21.223 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.164+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa23810b0f0 con 0x7fa2380ff650 2023-07-20T19:06:21.224 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.164+0000 7fa234ff9700 1 -- 172.21.15.61:0/3853685059 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fa22809b0f0 con 0x7fa2380ff650 2023-07-20T19:06:21.224 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:21.224 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T19:06:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1220 msgr2=0x7fa2240a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1220 0x7fa2240a36e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa220000c00 tx=0x7fa220009f10 comp rx=0 tx=0).stop 2023-07-20T19:06:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 msgr2=0x7fa2381953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 0x7fa2381953a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa22800beb0 tx=0x7fa22800dc30 comp rx=0 tx=0).stop 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 shutdown_connections 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1220 0x7fa2240a36e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2381011e0 0x7fa23819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2380ff650 0x7fa2381953a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 --2- 172.21.15.61:0/3853685059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa238100840 0x7fa2381958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 >> 172.21.15.61:0/3853685059 conn(0x7fa2380faba0 msgr2=0x7fa238103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 shutdown_connections 2023-07-20T19:06:21.228 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:21.166+0000 7fa23df03700 1 -- 172.21.15.61:0/3853685059 wait complete. 2023-07-20T19:06:21.228 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 4 2023-07-20T19:06:21.790 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:21.790 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:21.790 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:21.791 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:21.791 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:21.791 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:21.791 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:21.791 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:21 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3853685059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:22.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:22.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:21 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3853685059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:22.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:22.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:22.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:22.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:22.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:22.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:22.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:22.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:21 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3853685059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:22.554 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:06:22 smithi149 systemd[1]: Starting Ceph mds.e for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:22.888 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:06:22 smithi149 podman[153250]: 2023-07-20T19:06:22.888 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-mon[128885]: Deploying daemon mds.e on smithi149 2023-07-20T19:06:22.888 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-mon[128885]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:22.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.61:6836/2771268136,v1:172.21.15.61:6837/2771268136] up:boot 2023-07-20T19:06:22.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-mon[128885]: fsmap 4 up:standby 2023-07-20T19:06:22.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T19:06:23.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:22 smithi156 ceph-mon[129228]: Deploying daemon mds.e on smithi149 2023-07-20T19:06:23.167 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:22 smithi156 ceph-mon[129228]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:23.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:22 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.61:6836/2771268136,v1:172.21.15.61:6837/2771268136] up:boot 2023-07-20T19:06:23.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:22 smithi156 ceph-mon[129228]: fsmap 4 up:standby 2023-07-20T19:06:23.168 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:22 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T19:06:23.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:22 smithi061 ceph-mon[127454]: Deploying daemon mds.e on smithi149 2023-07-20T19:06:23.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:22 smithi061 ceph-mon[127454]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:23.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:22 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.61:6836/2771268136,v1:172.21.15.61:6837/2771268136] up:boot 2023-07-20T19:06:23.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:22 smithi061 ceph-mon[127454]: fsmap 4 up:standby 2023-07-20T19:06:23.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:22 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T19:06:23.239 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:23.304 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-mds[153290]: -- 172.21.15.149:0/2939063104 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56351f6d71e0 con 0x56351f6ac800 2023-07-20T19:06:23.305 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:06:22 smithi149 bash[153250]: a340c38870edd06d1a574ec53b44e61c3ef1c4a0aaf59f61a9bfee760e2d18b0 2023-07-20T19:06:23.305 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:06:22 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-e[153266]: starting mds.e at 2023-07-20T19:06:23.305 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:06:22 smithi149 systemd[1]: Started Ceph mds.e for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:23.379 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 -- 172.21.15.61:0/235113976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18100780 msgr2=0x7fce18100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 --2- 172.21.15.61:0/235113976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18100780 0x7fce18100ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce080099c0 tx=0x7fce08009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 -- 172.21.15.61:0/235113976 shutdown_connections 2023-07-20T19:06:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 --2- 172.21.15.61:0/235113976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce18102310 0x7fce18108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 --2- 172.21.15.61:0/235113976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18100780 0x7fce18100ba0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 --2- 172.21.15.61:0/235113976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18101970 0x7fce18101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 -- 172.21.15.61:0/235113976 >> 172.21.15.61:0/235113976 conn(0x7fce180fbcb0 msgr2=0x7fce180fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:23.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 -- 172.21.15.61:0/235113976 shutdown_connections 2023-07-20T19:06:23.381 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.320+0000 7fce1d143700 1 -- 172.21.15.61:0/235113976 wait complete. 2023-07-20T19:06:23.382 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 Processor -- start 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 -- start start 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18100780 0x7fce18196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce18101970 0x7fce181969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 0x7fce1819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1819e000 con 0x7fce18102310 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1819e140 con 0x7fce18100780 2023-07-20T19:06:23.383 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1d143700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1819e2d0 con 0x7fce18101970 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce1759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 0x7fce1819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.321+0000 7fce16d9d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18100780 0x7fce18196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 0x7fce1819da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58260/0 (socket says 172.21.15.61:58260) 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce16d9d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18100780 0x7fce18196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.61:58408/0 (socket says 172.21.15.61:58408) 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce0ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce18101970 0x7fce181969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 -- 172.21.15.61:0/116830371 learned_addr learned my addr 172.21.15.61:0/116830371 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 -- 172.21.15.61:0/116830371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce18101970 msgr2=0x7fce181969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce18101970 0x7fce181969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 -- 172.21.15.61:0/116830371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18100780 msgr2=0x7fce18196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18100780 0x7fce18196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 -- 172.21.15.61:0/116830371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce080095f0 con 0x7fce18102310 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1759e700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 0x7fce1819da30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fce1000ef30 tx=0x7fce1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce10018070 con 0x7fce18102310 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce1000f040 con 0x7fce18102310 2023-07-20T19:06:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1819e550 con 0x7fce18102310 2023-07-20T19:06:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1000cd00 con 0x7fce18102310 2023-07-20T19:06:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.322+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1819ea60 con 0x7fce18102310 2023-07-20T19:06:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.323+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce18107d10 con 0x7fce18102310 2023-07-20T19:06:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.324+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fce10003a00 con 0x7fce18102310 2023-07-20T19:06:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.325+0000 7fce14d99700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fce040a1220 0x7fce040a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.325+0000 7fce16d9d700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fce040a1220 0x7fce040a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.325+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fce100cc8f0 con 0x7fce18102310 2023-07-20T19:06:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.326+0000 7fce16d9d700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fce040a1220 0x7fce040a36e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fce08009990 tx=0x7fce08005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:23.388 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.329+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce10096570 con 0x7fce18102310 2023-07-20T19:06:23.632 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.573+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce1802c970 con 0x7fce18102310 2023-07-20T19:06:23.632 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.573+0000 7fce14d99700 1 -- 172.21.15.61:0/116830371 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7fce10099b90 con 0x7fce18102310 2023-07-20T19:06:23.633 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:23.633 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T19:06:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fce040a1220 msgr2=0x7fce040a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fce040a1220 0x7fce040a36e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fce08009990 tx=0x7fce08005fb0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 msgr2=0x7fce1819da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 0x7fce1819da30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fce1000ef30 tx=0x7fce1000c3a0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 shutdown_connections 2023-07-20T19:06:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fce040a1220 0x7fce040a36e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce18102310 0x7fce1819da30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce18100780 0x7fce18196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 --2- 172.21.15.61:0/116830371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce18101970 0x7fce181969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.575+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 >> 172.21.15.61:0/116830371 conn(0x7fce180fbcb0 msgr2=0x7fce18104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.576+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 shutdown_connections 2023-07-20T19:06:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:23.576+0000 7fce1d143700 1 -- 172.21.15.61:0/116830371 wait complete. 2023-07-20T19:06:23.637 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2023-07-20T19:06:24.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:24.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:24.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:24.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: Deploying daemon mds.f on smithi156 2023-07-20T19:06:24.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/116830371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:24.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.149:6834/725411194,v1:172.21.15.149:6835/725411194] up:boot 2023-07-20T19:06:24.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: fsmap 5 up:standby 2023-07-20T19:06:24.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:23 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T19:06:24.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:24.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:24.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:24.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: Deploying daemon mds.f on smithi156 2023-07-20T19:06:24.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/116830371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:24.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.149:6834/725411194,v1:172.21.15.149:6835/725411194] up:boot 2023-07-20T19:06:24.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: fsmap 5 up:standby 2023-07-20T19:06:24.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:23 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T19:06:24.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:24.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: Deploying daemon mds.f on smithi156 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/116830371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.149:6834/725411194,v1:172.21.15.149:6835/725411194] up:boot 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: fsmap 5 up:standby 2023-07-20T19:06:24.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:23 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T19:06:24.668 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:06:24 smithi156 systemd[1]: Starting Ceph mds.f for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:25.161 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:24 smithi156 ceph-mon[129228]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:25.162 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:06:24 smithi156 podman[154812]: 2023-07-20T19:06:25.162 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:06:25 smithi156 bash[154812]: 0a1a50900cb72c224d700e90cde9743c180e2f747f35d15dc7edb20ff3c326c4 2023-07-20T19:06:25.162 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:06:25 smithi156 ceph-mds[154853]: -- 172.21.15.156:0/3912867418 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ad434771e0 con 0x55ad4344c800 2023-07-20T19:06:25.163 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:06:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-f[154828]: starting mds.f at 2023-07-20T19:06:25.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:24 smithi149 ceph-mon[128885]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:25.417 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:06:25 smithi156 systemd[1]: Started Ceph mds.f for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:25.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:24 smithi061 ceph-mon[127454]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:25.648 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:25.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 -- 172.21.15.61:0/3845112874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8101230 msgr2=0x7f65f8107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:25.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 --2- 172.21.15.61:0/3845112874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8101230 0x7f65f8107ae0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f65f000b0d0 tx=0x7f65f000b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 -- 172.21.15.61:0/3845112874 shutdown_connections 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 --2- 172.21.15.61:0/3845112874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8101230 0x7f65f8107ae0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 --2- 172.21.15.61:0/3845112874 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65f80ff6a0 0x7f65f80ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 --2- 172.21.15.61:0/3845112874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100890 0x7f65f8100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 -- 172.21.15.61:0/3845112874 >> 172.21.15.61:0/3845112874 conn(0x7f65f80fabf0 msgr2=0x7f65f80fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 -- 172.21.15.61:0/3845112874 shutdown_connections 2023-07-20T19:06:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 -- 172.21.15.61:0/3845112874 wait complete. 2023-07-20T19:06:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 Processor -- start 2023-07-20T19:06:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.731+0000 7f65fda22700 1 -- start start 2023-07-20T19:06:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f80ff6a0 0x7f65f8069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 0x7f65f819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65f8101230 0x7f65f819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f819e030 con 0x7f65f8100890 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f819e1a0 con 0x7f65f8101230 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f819e360 con 0x7f65f80ff6a0 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 0x7f65f819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 0x7f65f819b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58270/0 (socket says 172.21.15.61:58270) 2023-07-20T19:06:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f80ff6a0 0x7f65f8069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 -- 172.21.15.61:0/3346867670 learned_addr learned my addr 172.21.15.61:0/3346867670 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f77fe700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65f8101230 0x7f65f819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 -- 172.21.15.61:0/3346867670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f80ff6a0 msgr2=0x7f65f8069700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f80ff6a0 0x7f65f8069700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 -- 172.21.15.61:0/3346867670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65f8101230 msgr2=0x7f65f819d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65f8101230 0x7f65f819d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 -- 172.21.15.61:0/3346867670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0009d00 con 0x7f65f8100890 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65f67fc700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 0x7f65f819b250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00e9c0 tx=0x7f65ec00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0097b0 con 0x7f65f8100890 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f819e640 con 0x7f65f8100890 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.732+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65ec004cf0 con 0x7f65f8100890 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.733+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec005600 con 0x7f65f8100890 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.733+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f819eb50 con 0x7f65f8100890 2023-07-20T19:06:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.733+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f81064e0 con 0x7f65f8100890 2023-07-20T19:06:25.797 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.735+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f65ec00cc60 con 0x7f65f8100890 2023-07-20T19:06:25.797 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.736+0000 7f65fca20700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f65e00a11a0 0x7f65e00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.736+0000 7f65f6ffd700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f65e00a11a0 0x7f65e00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.736+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f65ec014070 con 0x7f65f8100890 2023-07-20T19:06:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.736+0000 7f65f6ffd700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f65e00a11a0 0x7f65e00a3660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f65f81006f0 tx=0x7f65e800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.739+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f65ec096500 con 0x7f65f8100890 2023-07-20T19:06:26.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.986+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65f802c970 con 0x7f65f8100890 2023-07-20T19:06:26.045 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.986+0000 7f65fca20700 1 -- 172.21.15.61:0/3346867670 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7f65ec016330 con 0x7f65f8100890 2023-07-20T19:06:26.046 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:26.046 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-20T19:06:26.047 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f65e00a11a0 msgr2=0x7f65e00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:26.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f65e00a11a0 0x7f65e00a3660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f65f81006f0 tx=0x7f65e800a3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:26.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 msgr2=0x7f65f819b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:26.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 0x7f65f819b250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00e9c0 tx=0x7f65ec00ecd0 comp rx=0 tx=0).stop 2023-07-20T19:06:26.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 shutdown_connections 2023-07-20T19:06:26.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f65e00a11a0 0x7f65e00a3660 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:26.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65f8101230 0x7f65f819d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:26.049 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f80ff6a0 0x7f65f8069700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:26.049 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 --2- 172.21.15.61:0/3346867670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65f8100890 0x7f65f819b250 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:26.049 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 >> 172.21.15.61:0/3346867670 conn(0x7f65f80fabf0 msgr2=0x7f65f80fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:26.049 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 shutdown_connections 2023-07-20T19:06:26.049 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:25.988+0000 7f65fda22700 1 -- 172.21.15.61:0/3346867670 wait complete. 2023-07-20T19:06:26.049 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2023-07-20T19:06:26.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.470 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: Deploying daemon mds.c on smithi156 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6834/1691160499,v1:172.21.15.156:6835/1691160499] up:boot 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: fsmap 6 up:standby 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T19:06:26.471 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:26 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3346867670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:26.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:26.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:26.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:26.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: Deploying daemon mds.c on smithi156 2023-07-20T19:06:26.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6834/1691160499,v1:172.21.15.156:6835/1691160499] up:boot 2023-07-20T19:06:26.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: fsmap 6 up:standby 2023-07-20T19:06:26.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T19:06:26.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:26 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3346867670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:26.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: Deploying daemon mds.c on smithi156 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6834/1691160499,v1:172.21.15.156:6835/1691160499] up:boot 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: fsmap 6 up:standby 2023-07-20T19:06:26.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T19:06:26.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:26 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3346867670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:26.917 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:06:26 smithi156 systemd[1]: Starting Ceph mds.c for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:27.312 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:27 smithi156 ceph-mon[129228]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:27.312 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:06:26 smithi156 podman[155158]: 2023-07-20T19:06:27.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:27 smithi149 ceph-mon[128885]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:27.667 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:06:27 smithi156 ceph-mds[155198]: -- 172.21.15.156:0/621629960 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558670e8b1e0 con 0x558670e61000 2023-07-20T19:06:27.668 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:06:27 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-c[155174]: starting mds.c at 2023-07-20T19:06:27.668 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:06:27 smithi156 bash[155158]: ee3829f4ad76593781defceae9ef7db72ccd5fbe38fa6907442ef627cb64303a 2023-07-20T19:06:27.668 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:06:27 smithi156 systemd[1]: Started Ceph mds.c for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:27.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:27 smithi061 ceph-mon[127454]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:28.061 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:28.202 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 -- 172.21.15.61:0/1571867232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60100890 msgr2=0x7f4e60100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 --2- 172.21.15.61:0/1571867232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60100890 0x7f4e60100d10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4e54009990 tx=0x7f4e54009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 -- 172.21.15.61:0/1571867232 shutdown_connections 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 --2- 172.21.15.61:0/1571867232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e60101250 0x7f4e60107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 --2- 172.21.15.61:0/1571867232 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 0x7f4e600fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 --2- 172.21.15.61:0/1571867232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60100890 0x7f4e60100d10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 -- 172.21.15.61:0/1571867232 >> 172.21.15.61:0/1571867232 conn(0x7f4e600fa330 msgr2=0x7f4e600fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:28.203 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 -- 172.21.15.61:0/1571867232 shutdown_connections 2023-07-20T19:06:28.204 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.143+0000 7f4e67678700 1 -- 172.21.15.61:0/1571867232 wait complete. 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 Processor -- start 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 -- start start 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 0x7f4e601038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e60100890 0x7f4e60101f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 0x7f4e60102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60102c10 con 0x7f4e60101250 2023-07-20T19:06:28.206 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60102d80 con 0x7f4e600ffb80 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e67678700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e601a0430 con 0x7f4e60100890 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65c15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 0x7f4e60102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65414700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 0x7f4e601038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65c15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 0x7f4e60102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58286/0 (socket says 172.21.15.61:58286) 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65c15700 1 -- 172.21.15.61:0/2038166445 learned_addr learned my addr 172.21.15.61:0/2038166445 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e64c13700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e60100890 0x7f4e60101f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65414700 1 -- 172.21.15.61:0/2038166445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e60100890 msgr2=0x7f4e60101f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65414700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e60100890 0x7f4e60101f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65414700 1 -- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 msgr2=0x7f4e60102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65414700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 0x7f4e60102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65414700 1 -- 172.21.15.61:0/2038166445 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e540095f0 con 0x7f4e600ffb80 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.144+0000 7f4e65c15700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 0x7f4e60102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.145+0000 7f4e65414700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 0x7f4e601038f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000eee0 tx=0x7f4e5000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.145+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50018070 con 0x7f4e600ffb80 2023-07-20T19:06:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.145+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e601a06b0 con 0x7f4e600ffb80 2023-07-20T19:06:28.209 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.145+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e5000f040 con 0x7f4e600ffb80 2023-07-20T19:06:28.209 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.145+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50004280 con 0x7f4e600ffb80 2023-07-20T19:06:28.209 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.145+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e601a0bc0 con 0x7f4e600ffb80 2023-07-20T19:06:28.209 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.146+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e60107730 con 0x7f4e600ffb80 2023-07-20T19:06:28.210 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.147+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4e5000ca70 con 0x7f4e600ffb80 2023-07-20T19:06:28.210 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.148+0000 7f4e5e7fc700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f4e480a11a0 0x7f4e480a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:28.210 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.148+0000 7f4e64c13700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f4e480a11a0 0x7f4e480a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:28.210 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.148+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e500cc5a0 con 0x7f4e600ffb80 2023-07-20T19:06:28.210 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.148+0000 7f4e64c13700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f4e480a11a0 0x7f4e480a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006010 tx=0x7f4e5400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:28.211 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.151+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4e50096220 con 0x7f4e600ffb80 2023-07-20T19:06:28.457 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.398+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e6002cd90 con 0x7f4e600ffb80 2023-07-20T19:06:28.458 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.398+0000 7f4e5e7fc700 1 -- 172.21.15.61:0/2038166445 <== mon.2 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7f4e50099840 con 0x7f4e600ffb80 2023-07-20T19:06:28.458 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:28.458 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4080706577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4080706577},{"type":"v1","addr":"172.21.15.156:6837","nonce":4080706577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T19:06:28.460 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.400+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f4e480a11a0 msgr2=0x7f4e480a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:28.460 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.400+0000 7f4e67678700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f4e480a11a0 0x7f4e480a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006010 tx=0x7f4e5400b260 comp rx=0 tx=0).stop 2023-07-20T19:06:28.460 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.400+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 msgr2=0x7f4e601038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.400+0000 7f4e67678700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 0x7f4e601038f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000eee0 tx=0x7f4e5000c3a0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 shutdown_connections 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f4e480a11a0 0x7f4e480a3660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e60101250 0x7f4e60102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e600ffb80 0x7f4e601038f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 --2- 172.21.15.61:0/2038166445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e60100890 0x7f4e60101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 >> 172.21.15.61:0/2038166445 conn(0x7f4e600fa330 msgr2=0x7f4e600fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 shutdown_connections 2023-07-20T19:06:28.462 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:28.401+0000 7f4e67678700 1 -- 172.21.15.61:0/2038166445 wait complete. 2023-07-20T19:06:28.462 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 8 2023-07-20T19:06:28.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: Deploying daemon mds.h on smithi149 2023-07-20T19:06:28.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6836/4080706577,v1:172.21.15.156:6837/4080706577] up:boot 2023-07-20T19:06:28.669 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: fsmap 7 up:standby 2023-07-20T19:06:28.669 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:28 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T19:06:28.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:28.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:28.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:28.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: Deploying daemon mds.h on smithi149 2023-07-20T19:06:28.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6836/4080706577,v1:172.21.15.156:6837/4080706577] up:boot 2023-07-20T19:06:28.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: fsmap 7 up:standby 2023-07-20T19:06:28.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:28 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T19:06:28.749 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.749 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.749 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:28.749 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:28.750 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:28.750 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:28.750 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: Deploying daemon mds.h on smithi149 2023-07-20T19:06:28.750 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6836/4080706577,v1:172.21.15.156:6837/4080706577] up:boot 2023-07-20T19:06:28.750 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: fsmap 7 up:standby 2023-07-20T19:06:28.750 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:28 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T19:06:29.304 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:06:29 smithi149 systemd[1]: Starting Ceph mds.h for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:29.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:29 smithi156 ceph-mon[129228]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:29.667 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:29 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2038166445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:29.683 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:06:29 smithi149 podman[153599]: 2023-07-20T19:06:29.684 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:29 smithi149 ceph-mon[128885]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:29.684 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:29 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2038166445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:29.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:29 smithi061 ceph-mon[127454]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:29.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:29 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2038166445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:30.054 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:06:29 smithi149 ceph-mds[153639]: -- 172.21.15.149:0/3161964424 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d271ef1e0 con 0x557d271c4800 2023-07-20T19:06:30.054 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:06:29 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-h[153615]: starting mds.h at 2023-07-20T19:06:30.055 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:06:29 smithi149 bash[153599]: 94784f2bf0b6478d3475c6d67c68fa0f981df11b27732d3ca8a7eac868a61fe2 2023-07-20T19:06:30.055 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:06:29 smithi149 systemd[1]: Started Ceph mds.h for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:30.473 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.553+0000 7f41ae5ce700 1 -- 172.21.15.61:0/3505884519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8102300 msgr2=0x7f41a8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.553+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/3505884519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8102300 0x7f41a8108bb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41a000b0d0 tx=0x7f41a000b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.553+0000 7f41ae5ce700 1 -- 172.21.15.61:0/3505884519 shutdown_connections 2023-07-20T19:06:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.553+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/3505884519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8102300 0x7f41a8108bb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/3505884519 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41a8100770 0x7f41a8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/3505884519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41a8101960 0x7f41a8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 -- 172.21.15.61:0/3505884519 >> 172.21.15.61:0/3505884519 conn(0x7f41a80fbcc0 msgr2=0x7f41a80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 -- 172.21.15.61:0/3505884519 shutdown_connections 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 -- 172.21.15.61:0/3505884519 wait complete. 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 Processor -- start 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.554+0000 7f41ae5ce700 1 -- start start 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41a8100770 0x7f41a8074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 0x7f41a8072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41a8102300 0x7f41a80730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a8073840 con 0x7f41a8101960 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80739b0 con 0x7f41a8100770 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81a0530 con 0x7f41a8102300 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 0x7f41a8072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 0x7f41a8072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58298/0 (socket says 172.21.15.61:58298) 2023-07-20T19:06:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 -- 172.21.15.61:0/2771881973 learned_addr learned my addr 172.21.15.61:0/2771881973 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41acb6b700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41a8102300 0x7f41a80730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a7fff700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41a8100770 0x7f41a8074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 -- 172.21.15.61:0/2771881973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41a8102300 msgr2=0x7f41a80730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41a8102300 0x7f41a80730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 -- 172.21.15.61:0/2771881973 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41a8100770 msgr2=0x7f41a8074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41a8100770 0x7f41a8074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 -- 172.21.15.61:0/2771881973 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0009d00 con 0x7f41a8101960 2023-07-20T19:06:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a77fe700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 0x7f41a8072b60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f419c00e9a0 tx=0x7f419c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c009790 con 0x7f41a8101960 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f419c010430 con 0x7f41a8101960 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a81a0810 con 0x7f41a8101960 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c018680 con 0x7f41a8101960 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.555+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a81a0d70 con 0x7f41a8101960 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.556+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a810bf90 con 0x7f41a8101960 2023-07-20T19:06:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.558+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f419c0105a0 con 0x7f41a8101960 2023-07-20T19:06:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.558+0000 7f41a57fa700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f41900a1220 0x7f41900a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.559+0000 7f41a7fff700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f41900a1220 0x7f41900a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.559+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f419c014070 con 0x7f41a8101960 2023-07-20T19:06:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.559+0000 7f41a7fff700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f41900a1220 0x7f41900a36e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41a81017c0 tx=0x7f419800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.562+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f419c0963d0 con 0x7f41a8101960 2023-07-20T19:06:30.868 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.809+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41a8107580 con 0x7f41a8101960 2023-07-20T19:06:30.868 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.809+0000 7f41a57fa700 1 -- 172.21.15.61:0/2771881973 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7f419c0999f0 con 0x7f41a8101960 2023-07-20T19:06:30.869 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:30.869 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6837/2359948575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6836","nonce":2359948575},{"type":"v1","addr":"172.21.15.149:6837","nonce":2359948575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4080706577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4080706577},{"type":"v1","addr":"172.21.15.156:6837","nonce":4080706577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T19:06:30.870 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f41900a1220 msgr2=0x7f41900a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f41900a1220 0x7f41900a36e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41a81017c0 tx=0x7f419800a380 comp rx=0 tx=0).stop 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 msgr2=0x7f41a8072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 0x7f41a8072b60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f419c00e9a0 tx=0x7f419c00ecb0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 shutdown_connections 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f41900a1220 0x7f41900a36e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41a8102300 0x7f41a80730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.871 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41a8100770 0x7f41a8074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.872 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 --2- 172.21.15.61:0/2771881973 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41a8101960 0x7f41a8072b60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:30.872 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 >> 172.21.15.61:0/2771881973 conn(0x7f41a80fbcc0 msgr2=0x7f41a80fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:30.872 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 shutdown_connections 2023-07-20T19:06:30.872 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:30.811+0000 7f41ae5ce700 1 -- 172.21.15.61:0/2771881973 wait complete. 2023-07-20T19:06:30.872 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 9 2023-07-20T19:06:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: Deploying daemon mds.i on smithi156 2023-07-20T19:06:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.149:6836/2359948575,v1:172.21.15.149:6837/2359948575] up:boot 2023-07-20T19:06:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: fsmap 8 up:standby 2023-07-20T19:06:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:30 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T19:06:31.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:31.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: Deploying daemon mds.i on smithi156 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.149:6836/2359948575,v1:172.21.15.149:6837/2359948575] up:boot 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: fsmap 8 up:standby 2023-07-20T19:06:31.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:30 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T19:06:31.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:31.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: Deploying daemon mds.i on smithi156 2023-07-20T19:06:31.221 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.149:6836/2359948575,v1:172.21.15.149:6837/2359948575] up:boot 2023-07-20T19:06:31.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: fsmap 8 up:standby 2023-07-20T19:06:31.222 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:30 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T19:06:31.418 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:31 smithi156 systemd[1]: Starting Ceph mds.i for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:31.884 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:31 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2771881973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:31.885 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:31 smithi156 podman[155498]: 2023-07-20T19:06:31.886 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:31 smithi156 ceph-mds[155538]: -- 172.21.15.156:0/3476657096 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5634a178f1e0 con 0x5634a1764800 2023-07-20T19:06:31.886 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:31 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-i[155514]: starting mds.i at 2023-07-20T19:06:31.886 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:31 smithi156 bash[155498]: 6caa8284998cfcf8488ebc08730f0e2281d0d40bf8ab46338b2ba19b384d1a84 2023-07-20T19:06:32.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:31 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2771881973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:32.167 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:31 smithi156 systemd[1]: Started Ceph mds.i for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:32.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:31 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2771881973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:32.884 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:33.018 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.958+0000 7fab245c5700 1 -- 172.21.15.61:0/2628586935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c0ffb80 msgr2=0x7fab1c0fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:33.018 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.958+0000 7fab245c5700 1 --2- 172.21.15.61:0/2628586935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c0ffb80 0x7fab1c0fffa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab18009990 tx=0x7fab18009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.019 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 -- 172.21.15.61:0/2628586935 shutdown_connections 2023-07-20T19:06:33.019 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 --2- 172.21.15.61:0/2628586935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab1c101250 0x7fab1c107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.019 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 --2- 172.21.15.61:0/2628586935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c0ffb80 0x7fab1c0fffa0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.019 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 --2- 172.21.15.61:0/2628586935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab1c100890 0x7fab1c100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.019 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 -- 172.21.15.61:0/2628586935 >> 172.21.15.61:0/2628586935 conn(0x7fab1c0fa330 msgr2=0x7fab1c0fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:33.020 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 -- 172.21.15.61:0/2628586935 shutdown_connections 2023-07-20T19:06:33.020 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 -- 172.21.15.61:0/2628586935 wait complete. 2023-07-20T19:06:33.021 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 Processor -- start 2023-07-20T19:06:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.959+0000 7fab245c5700 1 -- start start 2023-07-20T19:06:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab1c0ffb80 0x7fab1c195360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 0x7fab1c1958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab1c101250 0x7fab1c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c19cee0 con 0x7fab1c100890 2023-07-20T19:06:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c19d020 con 0x7fab1c0ffb80 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c19d1b0 con 0x7fab1c101250 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 0x7fab1c1958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 0x7fab1c1958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58310/0 (socket says 172.21.15.61:58310) 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 -- 172.21.15.61:0/1297617061 learned_addr learned my addr 172.21.15.61:0/1297617061 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab22361700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab1c0ffb80 0x7fab1c195360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab22b62700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab1c101250 0x7fab1c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:33.023 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 -- 172.21.15.61:0/1297617061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab1c101250 msgr2=0x7fab1c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab1c101250 0x7fab1c19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 -- 172.21.15.61:0/1297617061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab1c0ffb80 msgr2=0x7fab1c195360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab1c0ffb80 0x7fab1c195360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 -- 172.21.15.61:0/1297617061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab180095f0 con 0x7fab1c100890 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab21b60700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 0x7fab1c1958a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0098f0 tx=0x7fab0c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c012070 con 0x7fab1c100890 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab0c0044e0 con 0x7fab1c100890 2023-07-20T19:06:33.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.960+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c0fe730 con 0x7fab1c100890 2023-07-20T19:06:33.025 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.961+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0c003df0 con 0x7fab1c100890 2023-07-20T19:06:33.025 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.961+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1c0fec40 con 0x7fab1c100890 2023-07-20T19:06:33.025 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.962+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1c10aea0 con 0x7fab1c100890 2023-07-20T19:06:33.025 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.963+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fab0c016030 con 0x7fab1c100890 2023-07-20T19:06:33.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.963+0000 7fab137fe700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab080a11a0 0x7fab080a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:33.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.963+0000 7fab22361700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab080a11a0 0x7fab080a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:33.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.963+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fab0c0cd580 con 0x7fab1c100890 2023-07-20T19:06:33.026 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.964+0000 7fab22361700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab080a11a0 0x7fab080a3660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fab18006010 tx=0x7fab18005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:33.027 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:32.968+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab0c097170 con 0x7fab1c100890 2023-07-20T19:06:33.174 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:33.175 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.175 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.175 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.175 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:33.175 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:33.175 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:33.176 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: Deploying daemon mds.g on smithi061 2023-07-20T19:06:33.176 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6838/1971202302,v1:172.21.15.156:6839/1971202302] up:boot 2023-07-20T19:06:33.176 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: fsmap 9 up:standby 2023-07-20T19:06:33.176 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:32 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T19:06:33.293 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.233+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab1c10b090 con 0x7fab1c100890 2023-07-20T19:06:33.293 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.234+0000 7fab137fe700 1 -- 172.21.15.61:0/1297617061 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7333 (secure 0 0 0) 0x7fab0c09a790 con 0x7fab1c100890 2023-07-20T19:06:33.294 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:33.294 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6837/2359948575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6836","nonce":2359948575},{"type":"v1","addr":"172.21.15.149:6837","nonce":2359948575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4080706577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4080706577},{"type":"v1","addr":"172.21.15.156:6837","nonce":4080706577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1971202302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1971202302},{"type":"v1","addr":"172.21.15.156:6839","nonce":1971202302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T19:06:33.295 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab080a11a0 msgr2=0x7fab080a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:33.295 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab080a11a0 0x7fab080a3660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fab18006010 tx=0x7fab18005960 comp rx=0 tx=0).stop 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 msgr2=0x7fab1c1958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 0x7fab1c1958a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0098f0 tx=0x7fab0c009c00 comp rx=0 tx=0).stop 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 shutdown_connections 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fab080a11a0 0x7fab080a3660 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab1c101250 0x7fab1c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab1c0ffb80 0x7fab1c195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 --2- 172.21.15.61:0/1297617061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c100890 0x7fab1c1958a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:33.297 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 >> 172.21.15.61:0/1297617061 conn(0x7fab1c0fa330 msgr2=0x7fab1c0fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:33.297 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 shutdown_connections 2023-07-20T19:06:33.297 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:33.236+0000 7fab245c5700 1 -- 172.21.15.61:0/1297617061 wait complete. 2023-07-20T19:06:33.297 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 10 2023-07-20T19:06:33.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:33.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: Deploying daemon mds.g on smithi061 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6838/1971202302,v1:172.21.15.156:6839/1971202302] up:boot 2023-07-20T19:06:33.305 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: fsmap 9 up:standby 2023-07-20T19:06:33.306 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:32 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T19:06:33.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:33.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: Deploying daemon mds.g on smithi061 2023-07-20T19:06:33.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6838/1971202302,v1:172.21.15.156:6839/1971202302] up:boot 2023-07-20T19:06:33.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: fsmap 9 up:standby 2023-07-20T19:06:33.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:32 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T19:06:33.752 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:06:33 smithi061 systemd[1]: Starting Ceph mds.g for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:34.063 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:33 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1297617061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:34.063 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:06:33 smithi061 podman[161520]: 2023-07-20T19:06:34.063 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:06:33 smithi061 bash[161520]: dbbffdecf0ef5ae2519b56156622218d2dd15a9d17b1e78c2f3927d5c4916107 2023-07-20T19:06:34.064 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:06:33 smithi061 ceph-mds[161560]: -- 172.21.15.61:0/1165998102 <== mon.0 v2:172.21.15.61:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55773e8651e0 con 0x55773e83ac00 2023-07-20T19:06:34.064 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:06:33 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-g[161536]: starting mds.g at 2023-07-20T19:06:34.304 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:33 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1297617061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:34.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:33 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1297617061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:34.470 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:06:34 smithi061 systemd[1]: Started Ceph mds.g for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:35.310 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:35.363 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:35.364 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: Deploying daemon mds.a on smithi061 2023-07-20T19:06:35.365 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.61:6838/1637414635,v1:172.21.15.61:6839/1637414635] up:boot 2023-07-20T19:06:35.365 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: fsmap 10 up:standby 2023-07-20T19:06:35.365 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:35 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T19:06:35.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:35.418 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: Deploying daemon mds.a on smithi061 2023-07-20T19:06:35.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.61:6838/1637414635,v1:172.21.15.61:6839/1637414635] up:boot 2023-07-20T19:06:35.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: fsmap 10 up:standby 2023-07-20T19:06:35.419 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:35 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T19:06:35.464 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/718023924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb00ff9a0 msgr2=0x7f1fb00ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:35.464 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/718023924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb00ffe20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009990 tx=0x7f1fa4009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.464 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/718023924 shutdown_connections 2023-07-20T19:06:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/718023924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fb0100360 0x7f1fb01007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/718023924 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff040 0x7f1fb00ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/718023924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb00ffe20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.404+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/718023924 >> 172.21.15.61:0/718023924 conn(0x7f1fb00faba0 msgr2=0x7f1fb00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.405+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/718023924 shutdown_connections 2023-07-20T19:06:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.405+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/718023924 wait complete. 2023-07-20T19:06:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.405+0000 7f1fb5fa5700 1 Processor -- start 2023-07-20T19:06:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 -- start start 2023-07-20T19:06:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fb00ff040 0x7f1fb01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb0198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 0x7f1fb019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01a0150 con 0x7f1fb0100360 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01a0290 con 0x7f1fb00ff9a0 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1fb5fa5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01a0420 con 0x7f1fb00ff040 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 0x7f1fb019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 0x7f1fb019fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58338/0 (socket says 172.21.15.61:58338) 2023-07-20T19:06:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 -- 172.21.15.61:0/3612404079 learned_addr learned my addr 172.21.15.61:0/3612404079 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faf7fe700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fb00ff040 0x7f1fb01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faeffd700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb0198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 -- 172.21.15.61:0/3612404079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fb00ff040 msgr2=0x7f1fb01985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fb00ff040 0x7f1fb01985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 -- 172.21.15.61:0/3612404079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff9a0 msgr2=0x7f1fb0198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb0198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 -- 172.21.15.61:0/3612404079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40095f0 con 0x7f1fb0100360 2023-07-20T19:06:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faeffd700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb0198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.406+0000 7f1faffff700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 0x7f1fb019fb80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800ada0 tx=0x7f1fa800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.407+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800f070 con 0x7f1fb0100360 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.407+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb01a0700 con 0x7f1fb0100360 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.407+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fa8007b30 con 0x7f1fb0100360 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.407+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80053e0 con 0x7f1fb0100360 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.407+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb01a0c10 con 0x7f1fb0100360 2023-07-20T19:06:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.408+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb0109e20 con 0x7f1fb0100360 2023-07-20T19:06:35.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.409+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1fa800c980 con 0x7f1fb0100360 2023-07-20T19:06:35.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.409+0000 7f1facff9700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1f980a11a0 0x7f1f980a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:35.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.410+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1fa80cbeb0 con 0x7f1fb0100360 2023-07-20T19:06:35.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.410+0000 7f1faf7fe700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1f980a11a0 0x7f1f980a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:35.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.410+0000 7f1faf7fe700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1f980a11a0 0x7f1f980a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005950 tx=0x7f1fa000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:35.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.413+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1fa8095a90 con 0x7f1fb0100360 2023-07-20T19:06:35.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:35.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: Deploying daemon mds.a on smithi061 2023-07-20T19:06:35.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.61:6838/1637414635,v1:172.21.15.61:6839/1637414635] up:boot 2023-07-20T19:06:35.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: fsmap 10 up:standby 2023-07-20T19:06:35.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:35 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T19:06:35.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.678+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fb002c970 con 0x7f1fb0100360 2023-07-20T19:06:35.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.679+0000 7f1facff9700 1 -- 172.21.15.61:0/3612404079 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8091 (secure 0 0 0) 0x7f1fa80990b0 con 0x7f1fb0100360 2023-07-20T19:06:35.739 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:35.739 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6839/1637414635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6838","nonce":1637414635},{"type":"v1","addr":"172.21.15.61:6839","nonce":1637414635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6837/2359948575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6836","nonce":2359948575},{"type":"v1","addr":"172.21.15.149:6837","nonce":2359948575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4080706577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4080706577},{"type":"v1","addr":"172.21.15.156:6837","nonce":4080706577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1971202302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1971202302},{"type":"v1","addr":"172.21.15.156:6839","nonce":1971202302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T19:06:35.741 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1f980a11a0 msgr2=0x7f1f980a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:35.741 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1f980a11a0 0x7f1f980a3660 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005950 tx=0x7f1fa000b410 comp rx=0 tx=0).stop 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 msgr2=0x7f1fb019fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 0x7f1fb019fb80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800ada0 tx=0x7f1fa800c330 comp rx=0 tx=0).stop 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 shutdown_connections 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1f980a11a0 0x7f1f980a3660 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fb0100360 0x7f1fb019fb80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fb00ff040 0x7f1fb01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 --2- 172.21.15.61:0/3612404079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1fb00ff9a0 0x7f1fb0198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:35.742 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 >> 172.21.15.61:0/3612404079 conn(0x7f1fb00faba0 msgr2=0x7f1fb00fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:35.743 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 shutdown_connections 2023-07-20T19:06:35.743 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:35.682+0000 7f1fb5fa5700 1 -- 172.21.15.61:0/3612404079 wait complete. 2023-07-20T19:06:35.743 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 11 2023-07-20T19:06:35.926 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:06:35 smithi061 systemd[1]: Starting Ceph mds.a for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:36.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:36 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3612404079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:36.220 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:06:35 smithi061 podman[161907]: 2023-07-20T19:06:36.220 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:06:36 smithi061 bash[161907]: 5b58318040b5b76e0cc9774cc602ebd71b2c39385ac175c668175df0608311df 2023-07-20T19:06:36.221 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:06:36 smithi061 ceph-mds[161948]: -- 172.21.15.61:0/109587828 <== mon.0 v2:172.21.15.61:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d7864631e0 con 0x55d786438800 2023-07-20T19:06:36.221 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:06:36 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-a[161923]: starting mds.a at 2023-07-20T19:06:36.417 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:36 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3612404079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:36.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:36 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3612404079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:36.720 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:06:36 smithi061 systemd[1]: Started Ceph mds.a for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:37.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:37.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:37.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:37.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:37.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: Deploying daemon mds.b on smithi149 2023-07-20T19:06:37.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.61:6840/3282516082,v1:172.21.15.61:6841/3282516082] up:boot 2023-07-20T19:06:37.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: fsmap 11 up:standby 2023-07-20T19:06:37.556 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:37 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T19:06:37.668 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:37.669 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.669 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.669 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.670 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:37.670 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:37.670 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:37.670 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: Deploying daemon mds.b on smithi149 2023-07-20T19:06:37.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.61:6840/3282516082,v1:172.21.15.61:6841/3282516082] up:boot 2023-07-20T19:06:37.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: fsmap 11 up:standby 2023-07-20T19:06:37.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:37 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T19:06:37.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:37.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:37.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T19:06:37.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T19:06:37.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:37.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: Deploying daemon mds.b on smithi149 2023-07-20T19:06:37.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.61:6840/3282516082,v1:172.21.15.61:6841/3282516082] up:boot 2023-07-20T19:06:37.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: fsmap 11 up:standby 2023-07-20T19:06:37.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:37 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T19:06:37.757 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:37.903 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 -- 172.21.15.61:0/500965097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec1011f0 msgr2=0x7fcaec107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:37.903 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/500965097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec1011f0 0x7fcaec107aa0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcae400b0d0 tx=0x7fcae400b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:37.903 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 -- 172.21.15.61:0/500965097 shutdown_connections 2023-07-20T19:06:37.904 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/500965097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec1011f0 0x7fcaec107aa0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:37.904 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/500965097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 0x7fcaec0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:37.904 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/500965097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcaec100850 0x7fcaec100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:37.904 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 -- 172.21.15.61:0/500965097 >> 172.21.15.61:0/500965097 conn(0x7fcaec0fabd0 msgr2=0x7fcaec0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:37.904 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 -- 172.21.15.61:0/500965097 shutdown_connections 2023-07-20T19:06:37.904 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.843+0000 7fcaf2b35700 1 -- 172.21.15.61:0/500965097 wait complete. 2023-07-20T19:06:37.906 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.844+0000 7fcaf2b35700 1 Processor -- start 2023-07-20T19:06:37.906 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.844+0000 7fcaf2b35700 1 -- start start 2023-07-20T19:06:37.906 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 0x7fcaec1953b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:37.906 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 0x7fcaec1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcaec1011f0 0x7fcaec19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec19cf30 con 0x7fcaec100850 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec19d070 con 0x7fcaec1011f0 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec19d200 con 0x7fcaec0ff660 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 0x7fcaec1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 0x7fcaec1958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58360/0 (socket says 172.21.15.61:58360) 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 -- 172.21.15.61:0/3756216188 learned_addr learned my addr 172.21.15.61:0/3756216188 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf10d2700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcaec1011f0 0x7fcaec19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf08d1700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 0x7fcaec1953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 -- 172.21.15.61:0/3756216188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 msgr2=0x7fcaec1953b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 0x7fcaec1953b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 -- 172.21.15.61:0/3756216188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcaec1011f0 msgr2=0x7fcaec19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcaec1011f0 0x7fcaec19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 -- 172.21.15.61:0/3756216188 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4009d00 con 0x7fcaec100850 2023-07-20T19:06:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf08d1700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 0x7fcaec1953b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaebfff700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 0x7fcaec1958f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcad800d980 tx=0x7fcad800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8004220 con 0x7fcaec100850 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcad8007640 con 0x7fcaec100850 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaec19d4e0 con 0x7fcaec100850 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.845+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8005610 con 0x7fcaec100850 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.846+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaec19d9f0 con 0x7fcaec100850 2023-07-20T19:06:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.847+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaec10aed0 con 0x7fcaec100850 2023-07-20T19:06:37.910 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.848+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcad8004750 con 0x7fcaec100850 2023-07-20T19:06:37.910 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.848+0000 7fcae9ffb700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcadc0a11a0 0x7fcadc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:37.910 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.849+0000 7fcaf08d1700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcadc0a11a0 0x7fcadc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.849+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcad80cc300 con 0x7fcaec100850 2023-07-20T19:06:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.849+0000 7fcaf08d1700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcadc0a11a0 0x7fcadc0a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcae0005fd0 tx=0x7fcae0005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:37.912 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:37.853+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcad8095ef0 con 0x7fcaec100850 2023-07-20T19:06:38.112 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:06:37 smithi149 systemd[1]: Starting Ceph mds.b for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:38.158 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.098+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaec10b0c0 con 0x7fcaec100850 2023-07-20T19:06:38.159 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.099+0000 7fcae9ffb700 1 -- 172.21.15.61:0/3756216188 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8849 (secure 0 0 0) 0x7fcaec10b0c0 con 0x7fcaec100850 2023-07-20T19:06:38.159 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:38.160 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6839/1637414635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6838","nonce":1637414635},{"type":"v1","addr":"172.21.15.61:6839","nonce":1637414635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6841/3282516082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6840","nonce":3282516082},{"type":"v1","addr":"172.21.15.61:6841","nonce":3282516082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6837/2359948575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6836","nonce":2359948575},{"type":"v1","addr":"172.21.15.149:6837","nonce":2359948575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4080706577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4080706577},{"type":"v1","addr":"172.21.15.156:6837","nonce":4080706577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1971202302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1971202302},{"type":"v1","addr":"172.21.15.156:6839","nonce":1971202302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T19:06:38.161 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcadc0a11a0 msgr2=0x7fcadc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcadc0a11a0 0x7fcadc0a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcae0005fd0 tx=0x7fcae0005e40 comp rx=0 tx=0).stop 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 msgr2=0x7fcaec1958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 0x7fcaec1958f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcad800d980 tx=0x7fcad800dc90 comp rx=0 tx=0).stop 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 shutdown_connections 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fcadc0a11a0 0x7fcadc0a3660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcaec1011f0 0x7fcaec19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaec0ff660 0x7fcaec1953b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 --2- 172.21.15.61:0/3756216188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcaec100850 0x7fcaec1958f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 >> 172.21.15.61:0/3756216188 conn(0x7fcaec0fabd0 msgr2=0x7fcaec0fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 shutdown_connections 2023-07-20T19:06:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:38.101+0000 7fcaf2b35700 1 -- 172.21.15.61:0/3756216188 wait complete. 2023-07-20T19:06:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2023-07-20T19:06:38.423 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:06:38 smithi149 podman[153939]: 2023-07-20T19:06:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:38 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3756216188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:38.804 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:38.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:38 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:38.805 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:06:38 smithi149 ceph-mds[153980]: -- 172.21.15.149:0/3688637870 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564c097ab1e0 con 0x564c09780c00 2023-07-20T19:06:38.805 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:06:38 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-b[153955]: starting mds.b at 2023-07-20T19:06:38.805 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:06:38 smithi149 bash[153939]: 8570309edb0e40da9808b0517a2acd87098f8ab1c7516a475f86af382ea528ea 2023-07-20T19:06:38.805 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:06:38 smithi149 systemd[1]: Started Ceph mds.b for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:38.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:38 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3756216188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:38.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:38.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:38 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:38 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3756216188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:38.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:38 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.854 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:39.855 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.149:6838/284468426,v1:172.21.15.149:6839/284468426] up:boot 2023-07-20T19:06:39.855 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: fsmap 12 up:standby 2023-07-20T19:06:39.856 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T19:06:39.856 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.856 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.857 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.857 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:39 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:39.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:39.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.149:6838/284468426,v1:172.21.15.149:6839/284468426] up:boot 2023-07-20T19:06:39.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: fsmap 12 up:standby 2023-07-20T19:06:39.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T19:06:39.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:39 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:39.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.149:6838/284468426,v1:172.21.15.149:6839/284468426] up:boot 2023-07-20T19:06:39.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: fsmap 12 up:standby 2023-07-20T19:06:39.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T19:06:39.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:39.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:39 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:40.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T19:06:40.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 -- 172.21.15.61:0/250126094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250102450 msgr2=0x7fa250108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 --2- 172.21.15.61:0/250126094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250102450 0x7fa250108d00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa24800b0a0 tx=0x7fa24800b3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.304 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 -- 172.21.15.61:0/250126094 shutdown_connections 2023-07-20T19:06:40.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 --2- 172.21.15.61:0/250126094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250102450 0x7fa250108d00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 --2- 172.21.15.61:0/250126094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2500fea40 0x7fa2500fee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 --2- 172.21.15.61:0/250126094 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa250101ab0 0x7fa250101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 -- 172.21.15.61:0/250126094 >> 172.21.15.61:0/250126094 conn(0x7fa2500fa330 msgr2=0x7fa2500fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:40.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 -- 172.21.15.61:0/250126094 shutdown_connections 2023-07-20T19:06:40.305 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.244+0000 7fa258446700 1 -- 172.21.15.61:0/250126094 wait complete. 2023-07-20T19:06:40.307 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 Processor -- start 2023-07-20T19:06:40.307 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 -- start start 2023-07-20T19:06:40.307 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2500fea40 0x7fa2501964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 0x7fa250196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa250102450 0x7fa25019da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25019e050 con 0x7fa250101ab0 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25019e190 con 0x7fa2500fea40 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa258446700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25019e320 con 0x7fa250102450 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.245+0000 7fa2559e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 0x7fa250196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.308 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 0x7fa250196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58372/0 (socket says 172.21.15.61:58372) 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 -- 172.21.15.61:0/84915985 learned_addr learned my addr 172.21.15.61:0/84915985 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2561e2700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2500fea40 0x7fa2501964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2569e3700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa250102450 0x7fa25019da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 -- 172.21.15.61:0/84915985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa250102450 msgr2=0x7fa25019da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa250102450 0x7fa25019da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 -- 172.21.15.61:0/84915985 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2500fea40 msgr2=0x7fa2501964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.309 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2500fea40 0x7fa2501964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 -- 172.21.15.61:0/84915985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248009d00 con 0x7fa250101ab0 2023-07-20T19:06:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2559e1700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 0x7fa250196a10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa24000d950 tx=0x7fa24000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24000f890 con 0x7fa250101ab0 2023-07-20T19:06:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa240010460 con 0x7fa250101ab0 2023-07-20T19:06:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25019e600 con 0x7fa250101ab0 2023-07-20T19:06:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24000e610 con 0x7fa250101ab0 2023-07-20T19:06:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.246+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25019eb60 con 0x7fa250101ab0 2023-07-20T19:06:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.247+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250107e00 con 0x7fa250101ab0 2023-07-20T19:06:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.248+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa2400108d0 con 0x7fa250101ab0 2023-07-20T19:06:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.249+0000 7fa2477fe700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa23c0a1220 0x7fa23c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.249+0000 7fa2561e2700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa23c0a1220 0x7fa23c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.249+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2400cc1b0 con 0x7fa250101ab0 2023-07-20T19:06:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.250+0000 7fa2561e2700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa23c0a1220 0x7fa23c0a36e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa250067610 tx=0x7fa24c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:40.314 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.255+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa240095da0 con 0x7fa250101ab0 2023-07-20T19:06:40.563 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.504+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa25002c970 con 0x7fa250101ab0 2023-07-20T19:06:40.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.505+0000 7fa2477fe700 1 -- 172.21.15.61:0/84915985 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9607 (secure 0 0 0) 0x7fa2400993c0 con 0x7fa250101ab0 2023-07-20T19:06:40.565 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:40.566 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6835/1326256186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6834","nonce":1326256186},{"type":"v1","addr":"172.21.15.61:6835","nonce":1326256186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6837/2771268136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6836","nonce":2771268136},{"type":"v1","addr":"172.21.15.61:6837","nonce":2771268136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6839/1637414635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6838","nonce":1637414635},{"type":"v1","addr":"172.21.15.61:6839","nonce":1637414635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6841/3282516082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6840","nonce":3282516082},{"type":"v1","addr":"172.21.15.61:6841","nonce":3282516082}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6833/1646752575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1646752575},{"type":"v1","addr":"172.21.15.149:6833","nonce":1646752575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/725411194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6834","nonce":725411194},{"type":"v1","addr":"172.21.15.149:6835","nonce":725411194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2641203790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2641203790},{"type":"v1","addr":"172.21.15.156:6833","nonce":2641203790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6837/2359948575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6836","nonce":2359948575},{"type":"v1","addr":"172.21.15.149:6837","nonce":2359948575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1691160499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1691160499},{"type":"v1","addr":"172.21.15.156:6835","nonce":1691160499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4080706577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4080706577},{"type":"v1","addr":"172.21.15.156:6837","nonce":4080706577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6839/284468426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6838","nonce":284468426},{"type":"v1","addr":"172.21.15.149:6839","nonce":284468426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1971202302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1971202302},{"type":"v1","addr":"172.21.15.156:6839","nonce":1971202302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T19:06:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa23c0a1220 msgr2=0x7fa23c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa23c0a1220 0x7fa23c0a36e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa250067610 tx=0x7fa24c00b410 comp rx=0 tx=0).stop 2023-07-20T19:06:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 msgr2=0x7fa250196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 0x7fa250196a10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa24000d950 tx=0x7fa24000dc60 comp rx=0 tx=0).stop 2023-07-20T19:06:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 shutdown_connections 2023-07-20T19:06:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa23c0a1220 0x7fa23c0a36e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa250102450 0x7fa25019da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2500fea40 0x7fa2501964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 --2- 172.21.15.61:0/84915985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa250101ab0 0x7fa250196a10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 >> 172.21.15.61:0/84915985 conn(0x7fa2500fa330 msgr2=0x7fa250104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 shutdown_connections 2023-07-20T19:06:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.507+0000 7fa258446700 1 -- 172.21.15.61:0/84915985 wait complete. 2023-07-20T19:06:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 13 2023-07-20T19:06:40.579 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T19:06:40.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 -- 172.21.15.61:0/1526605329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c103b60 msgr2=0x7f677c105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.711 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 --2- 172.21.15.61:0/1526605329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c103b60 0x7f677c105f50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6768009990 tx=0x7f6768009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 -- 172.21.15.61:0/1526605329 shutdown_connections 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 --2- 172.21.15.61:0/1526605329 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 0x7f677c1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 --2- 172.21.15.61:0/1526605329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f677c101230 0x7f677c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 --2- 172.21.15.61:0/1526605329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c103b60 0x7f677c105f50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 -- 172.21.15.61:0/1526605329 >> 172.21.15.61:0/1526605329 conn(0x7f677c0faba0 msgr2=0x7f677c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 -- 172.21.15.61:0/1526605329 shutdown_connections 2023-07-20T19:06:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.651+0000 7f6782b6b700 1 -- 172.21.15.61:0/1526605329 wait complete. 2023-07-20T19:06:40.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 Processor -- start 2023-07-20T19:06:40.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 -- start start 2023-07-20T19:06:40.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 0x7f677c196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.714 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f677c103b60 0x7f677c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 0x7f677c19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19dfa0 con 0x7f677c101230 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19e0e0 con 0x7f677c106490 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6782b6b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19e270 con 0x7f677c103b60 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6780907700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 0x7f677c196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6780907700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 0x7f677c196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58384/0 (socket says 172.21.15.61:58384) 2023-07-20T19:06:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6780907700 1 -- 172.21.15.61:0/4139589308 learned_addr learned my addr 172.21.15.61:0/4139589308 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f677bfff700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f677c103b60 0x7f677c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.652+0000 7f6781108700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 0x7f677c19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6780907700 1 -- 172.21.15.61:0/4139589308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f677c103b60 msgr2=0x7f677c196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6780907700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f677c103b60 0x7f677c196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6780907700 1 -- 172.21.15.61:0/4139589308 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 msgr2=0x7f677c19d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6780907700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 0x7f677c19d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6780907700 1 -- 172.21.15.61:0/4139589308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67680095f0 con 0x7f677c101230 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6781108700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 0x7f677c19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6780907700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 0x7f677c196420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f677000d950 tx=0x7f677000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770009940 con 0x7f677c101230 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6770010460 con 0x7f677c101230 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c19e550 con 0x7f677c101230 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677000f620 con 0x7f677c101230 2023-07-20T19:06:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.653+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c19eab0 con 0x7f677c101230 2023-07-20T19:06:40.718 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.654+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c0fc720 con 0x7f677c101230 2023-07-20T19:06:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.655+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f67700108d0 con 0x7f677c101230 2023-07-20T19:06:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.656+0000 7f6779ffb700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f676c0a1220 0x7f676c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.656+0000 7f677bfff700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f676c0a1220 0x7f676c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.656+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f67700cc310 con 0x7f677c101230 2023-07-20T19:06:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.656+0000 7f677bfff700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f676c0a1220 0x7f676c0a36e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f676800b5c0 tx=0x7f6768005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:40.720 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.660+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6770095e60 con 0x7f677c101230 2023-07-20T19:06:40.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:40 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/84915985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:40.937 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.877+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f677c02c970 con 0x7f677c101230 2023-07-20T19:06:40.938 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.878+0000 7f6779ffb700 1 -- 172.21.15.61:0/4139589308 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18408 (secure 0 0 0) 0x7f6770099480 con 0x7f677c101230 2023-07-20T19:06:40.938 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:40.939 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","created":"2023-07-20T18:59:47.496551+0000","modified":"2023-07-20T19:06:04.764327+0000","last_up_change":"2023-07-20T19:06:03.755886+0000","last_in_change":"2023-07-20T19:05:46.242910+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T19:02:56.142587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfae06af-18f4-487e-ae4b-8e8c9d36ef76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6803","nonce":1072462099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6805","nonce":1072462099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6809","nonce":1072462099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6807","nonce":1072462099}]},"public_addr":"172.21.15.61:6803/1072462099","cluster_addr":"172.21.15.61:6805/1072462099","heartbeat_back_addr":"172.21.15.61:6809/1072462099","heartbeat_front_addr":"172.21.15.61:6807/1072462099","state":["exists","up"]},{"osd":1,"uuid":"d3e2b1d4-abe5-4438-9196-6ef26230a071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6801","nonce":414280084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6803","nonce":414280084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6807","nonce":414280084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6805","nonce":414280084}]},"public_addr":"172.21.15.149:6801/414280084","cluster_addr":"172.21.15.149:6803/414280084","heartbeat_back_addr":"172.21.15.149:6807/414280084","heartbeat_front_addr":"172.21.15.149:6805/414280084","state":["exists","up"]},{"osd":2,"uuid":"34a03029-87a9-49e1-b262-32be5070df0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6801","nonce":738107812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6803","nonce":738107812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6807","nonce":738107812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6805","nonce":738107812}]},"public_addr":"172.21.15.156:6801/738107812","cluster_addr":"172.21.15.156:6803/738107812","heartbeat_back_addr":"172.21.15.156:6807/738107812","heartbeat_front_addr":"172.21.15.156:6805/738107812","state":["exists","up"]},{"osd":3,"uuid":"c5711133-530f-488a-9db4-2b8a61b7ea02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6811","nonce":1371683823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6813","nonce":1371683823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6817","nonce":1371683823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6815","nonce":1371683823}]},"public_addr":"172.21.15.61:6811/1371683823","cluster_addr":"172.21.15.61:6813/1371683823","heartbeat_back_addr":"172.21.15.61:6817/1371683823","heartbeat_front_addr":"172.21.15.61:6815/1371683823","state":["exists","up"]},{"osd":4,"uuid":"6062cdd3-5bf1-458a-8454-253627db736f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6809","nonce":3513915816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6811","nonce":3513915816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6815","nonce":3513915816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6813","nonce":3513915816}]},"public_addr":"172.21.15.149:6809/3513915816","cluster_addr":"172.21.15.149:6811/3513915816","heartbeat_back_addr":"172.21.15.149:6815/3513915816","heartbeat_front_addr":"172.21.15.149:6813/3513915816","state":["exists","up"]},{"osd":5,"uuid":"1fae1b15-43cb-40d9-ad90-73af4cc96197","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6809","nonce":211953844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6811","nonce":211953844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6815","nonce":211953844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6813","nonce":211953844}]},"public_addr":"172.21.15.156:6809/211953844","cluster_addr":"172.21.15.156:6811/211953844","heartbeat_back_addr":"172.21.15.156:6815/211953844","heartbeat_front_addr":"172.21.15.156:6813/211953844","state":["exists","up"]},{"osd":6,"uuid":"84316d15-3a51-432c-9d02-57270ac09769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6819","nonce":1770665297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6821","nonce":1770665297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6825","nonce":1770665297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6823","nonce":1770665297}]},"public_addr":"172.21.15.61:6819/1770665297","cluster_addr":"172.21.15.61:6821/1770665297","heartbeat_back_addr":"172.21.15.61:6825/1770665297","heartbeat_front_addr":"172.21.15.61:6823/1770665297","state":["exists","up"]},{"osd":7,"uuid":"705c1527-9342-402c-add4-4382e005dbfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6817","nonce":1095358817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6819","nonce":1095358817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6823","nonce":1095358817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6821","nonce":1095358817}]},"public_addr":"172.21.15.149:6817/1095358817","cluster_addr":"172.21.15.149:6819/1095358817","heartbeat_back_addr":"172.21.15.149:6823/1095358817","heartbeat_front_addr":"172.21.15.149:6821/1095358817","state":["exists","up"]},{"osd":8,"uuid":"6e790866-ee13-4c27-bbd4-ea90e853de1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6817","nonce":1602221626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6819","nonce":1602221626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6823","nonce":1602221626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6821","nonce":1602221626}]},"public_addr":"172.21.15.156:6817/1602221626","cluster_addr":"172.21.15.156:6819/1602221626","heartbeat_back_addr":"172.21.15.156:6823/1602221626","heartbeat_front_addr":"172.21.15.156:6821/1602221626","state":["exists","up"]},{"osd":9,"uuid":"ac0b69d5-ccea-470d-a8c8-b9a2f508af24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6827","nonce":2654664682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6829","nonce":2654664682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6833","nonce":2654664682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6831","nonce":2654664682}]},"public_addr":"172.21.15.61:6827/2654664682","cluster_addr":"172.21.15.61:6829/2654664682","heartbeat_back_addr":"172.21.15.61:6833/2654664682","heartbeat_front_addr":"172.21.15.61:6831/2654664682","state":["exists","up"]},{"osd":10,"uuid":"1a6e7d98-a3f2-4d81-843e-bca056ec748c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6825","nonce":1912645375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6827","nonce":1912645375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6831","nonce":1912645375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6829","nonce":1912645375}]},"public_addr":"172.21.15.149:6825/1912645375","cluster_addr":"172.21.15.149:6827/1912645375","heartbeat_back_addr":"172.21.15.149:6831/1912645375","heartbeat_front_addr":"172.21.15.149:6829/1912645375","state":["exists","up"]},{"osd":11,"uuid":"5bcb26b2-0789-4cf8-9803-98a6da33d5d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6825","nonce":893354887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6827","nonce":893354887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6831","nonce":893354887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6829","nonce":893354887}]},"public_addr":"172.21.15.156:6825/893354887","cluster_addr":"172.21.15.156:6827/893354887","heartbeat_back_addr":"172.21.15.156:6831/893354887","heartbeat_front_addr":"172.21.15.156:6829/893354887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:09.841610+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:31.306806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:52.704303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:13.230276+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:34.832690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:55.534516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:16.097468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:37.618136+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:58.685044+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:19.824839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:41.039703+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:06:02.038214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.61:6801/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6800/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/3344679989":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6801/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/170145058":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/575168883":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/2333936767":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/74416212":"2023-07-21T19:00:21.461577+0000","172.21.15.61:6800/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/2426181232":"2023-07-21T19:00:21.461577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T19:06:40.940 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f676c0a1220 msgr2=0x7f676c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.940 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f676c0a1220 0x7f676c0a36e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f676800b5c0 tx=0x7f6768005fd0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 msgr2=0x7f677c196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 0x7f677c196420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f677000d950 tx=0x7f677000dc60 comp rx=0 tx=0).stop 2023-07-20T19:06:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 shutdown_connections 2023-07-20T19:06:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f676c0a1220 0x7f676c0a36e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f677c106490 0x7f677c19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f677c101230 0x7f677c196420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.942 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 --2- 172.21.15.61:0/4139589308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f677c103b60 0x7f677c196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:40.942 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 >> 172.21.15.61:0/4139589308 conn(0x7f677c0faba0 msgr2=0x7f677c101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:40.942 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 shutdown_connections 2023-07-20T19:06:40.942 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:40.881+0000 7f6782b6b700 1 -- 172.21.15.61:0/4139589308 wait complete. 2023-07-20T19:06:40.952 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:40 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/84915985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:40.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T19:02:56.142587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T19:06:40.954 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T19:06:41.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:40 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/84915985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T19:06:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 -- 172.21.15.61:0/915994013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba40ff9c0 msgr2=0x7f3ba40ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/915994013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba40ff9c0 0x7f3ba40ffe40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3b980099c0 tx=0x7f3b98009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 -- 172.21.15.61:0/915994013 shutdown_connections 2023-07-20T19:06:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/915994013 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ba4100380 0x7f3ba4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/915994013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ba40ff060 0x7f3ba40ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/915994013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba40ff9c0 0x7f3ba40ffe40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 -- 172.21.15.61:0/915994013 >> 172.21.15.61:0/915994013 conn(0x7f3ba40fac00 msgr2=0x7f3ba40fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 -- 172.21.15.61:0/915994013 shutdown_connections 2023-07-20T19:06:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.027+0000 7f3ba8d08700 1 -- 172.21.15.61:0/915994013 wait complete. 2023-07-20T19:06:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 Processor -- start 2023-07-20T19:06:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 -- start start 2023-07-20T19:06:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ba40ff060 0x7f3ba4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ba40ff9c0 0x7f3ba4198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 0x7f3ba419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.094 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41a01c0 con 0x7f3ba4100380 2023-07-20T19:06:41.094 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41a0300 con 0x7f3ba40ff9c0 2023-07-20T19:06:41.094 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba8d08700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41a0490 con 0x7f3ba40ff060 2023-07-20T19:06:41.095 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 0x7f3ba419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.095 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ba40ff060 0x7f3ba4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.095 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 0x7f3ba419fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58402/0 (socket says 172.21.15.61:58402) 2023-07-20T19:06:41.095 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba2d9d700 1 -- 172.21.15.61:0/3743304397 learned_addr learned my addr 172.21.15.61:0/3743304397 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:41.096 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.028+0000 7f3ba1d9b700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ba40ff9c0 0x7f3ba4198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.096 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba2d9d700 1 -- 172.21.15.61:0/3743304397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ba40ff060 msgr2=0x7f3ba4198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.096 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba2d9d700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ba40ff060 0x7f3ba4198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.097 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba2d9d700 1 -- 172.21.15.61:0/3743304397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ba40ff9c0 msgr2=0x7f3ba4198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.097 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba2d9d700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ba40ff9c0 0x7f3ba4198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.097 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba2d9d700 1 -- 172.21.15.61:0/3743304397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94009520 con 0x7f3ba4100380 2023-07-20T19:06:41.097 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba2d9d700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 0x7f3ba419fbf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c370 tx=0x7f3b9c00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:41.098 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c017070 con 0x7f3ba4100380 2023-07-20T19:06:41.098 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b9c00f040 con 0x7f3ba4100380 2023-07-20T19:06:41.098 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c00e050 con 0x7f3ba4100380 2023-07-20T19:06:41.099 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c009c40 con 0x7f3ba4100380 2023-07-20T19:06:41.099 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.029+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b980095f0 con 0x7f3ba4100380 2023-07-20T19:06:41.099 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.030+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba4109e40 con 0x7f3ba4100380 2023-07-20T19:06:41.101 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.031+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3b9c00cb60 con 0x7f3ba4100380 2023-07-20T19:06:41.101 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.032+0000 7f3b937fe700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3b8c0a1150 0x7f3b8c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.101 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.032+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3b9c0cc4c0 con 0x7f3ba4100380 2023-07-20T19:06:41.101 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.032+0000 7f3ba259c700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3b8c0a1150 0x7f3b8c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.102 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.032+0000 7f3ba259c700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3b8c0a1150 0x7f3b8c0a3610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009df0 tx=0x7f3b94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:41.102 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.036+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b9c0960b0 con 0x7f3ba4100380 2023-07-20T19:06:41.319 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.259+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ba402ce00 con 0x7f3ba4100380 2023-07-20T19:06:41.320 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.260+0000 7f3b937fe700 1 -- 172.21.15.61:0/3743304397 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b9c0996d0 con 0x7f3ba4100380 2023-07-20T19:06:41.320 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2023-07-20T19:06:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3b8c0a1150 msgr2=0x7f3b8c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3b8c0a1150 0x7f3b8c0a3610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009df0 tx=0x7f3b94009260 comp rx=0 tx=0).stop 2023-07-20T19:06:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 msgr2=0x7f3ba419fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 0x7f3ba419fbf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c370 tx=0x7f3b9c00c680 comp rx=0 tx=0).stop 2023-07-20T19:06:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 shutdown_connections 2023-07-20T19:06:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3b8c0a1150 0x7f3b8c0a3610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ba4100380 0x7f3ba419fbf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ba40ff060 0x7f3ba4198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 --2- 172.21.15.61:0/3743304397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ba40ff9c0 0x7f3ba4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 >> 172.21.15.61:0/3743304397 conn(0x7f3ba40fac00 msgr2=0x7f3ba40fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 shutdown_connections 2023-07-20T19:06:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.262+0000 7f3ba8d08700 1 -- 172.21.15.61:0/3743304397 wait complete. 2023-07-20T19:06:41.334 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T19:06:41.467 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.407+0000 7fe9cf566700 1 -- 172.21.15.61:0/1294164730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 msgr2=0x7fe9c8107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.407+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1294164730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c8107ab0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000b0d0 tx=0x7fe9c000b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- 172.21.15.61:0/1294164730 shutdown_connections 2023-07-20T19:06:41.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1294164730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c8107ab0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1294164730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c80ff670 0x7fe9c80ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1294164730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c8100860 0x7fe9c8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.468 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- 172.21.15.61:0/1294164730 >> 172.21.15.61:0/1294164730 conn(0x7fe9c80faba0 msgr2=0x7fe9c80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:41.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- 172.21.15.61:0/1294164730 shutdown_connections 2023-07-20T19:06:41.469 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- 172.21.15.61:0/1294164730 wait complete. 2023-07-20T19:06:41.471 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 Processor -- start 2023-07-20T19:06:41.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- start start 2023-07-20T19:06:41.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c80ff670 0x7fe9c8196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.472 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c8100860 0x7fe9c8196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c819e0a0 con 0x7fe9c8101200 2023-07-20T19:06:41.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c819e1e0 con 0x7fe9c80ff670 2023-07-20T19:06:41.473 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.408+0000 7fe9cf566700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c819e370 con 0x7fe9c8100860 2023-07-20T19:06:41.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cd302700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c80ff670 0x7fe9c8196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.474 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58416/0 (socket says 172.21.15.61:58416) 2023-07-20T19:06:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9ccb01700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c8100860 0x7fe9c8196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 -- 172.21.15.61:0/1016703890 learned_addr learned my addr 172.21.15.61:0/1016703890 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 -- 172.21.15.61:0/1016703890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c8100860 msgr2=0x7fe9c8196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c8100860 0x7fe9c8196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 -- 172.21.15.61:0/1016703890 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c80ff670 msgr2=0x7fe9c8196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c80ff670 0x7fe9c8196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 -- 172.21.15.61:0/1016703890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0009d00 con 0x7fe9c8101200 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cd302700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c80ff670 0x7fe9c8196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cdb03700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c819dad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0008eb0 tx=0x7fe9c0008ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c000e040 con 0x7fe9c8101200 2023-07-20T19:06:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9c000bdb0 con 0x7fe9c8101200 2023-07-20T19:06:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c819e5f0 con 0x7fe9c8101200 2023-07-20T19:06:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c001d470 con 0x7fe9c8101200 2023-07-20T19:06:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.409+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c819eac0 con 0x7fe9c8101200 2023-07-20T19:06:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.410+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c8107660 con 0x7fe9c8101200 2023-07-20T19:06:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.412+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe9c0005000 con 0x7fe9c8101200 2023-07-20T19:06:41.478 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.413+0000 7fe9be7fc700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe9b40a1220 0x7fe9b40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.478 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.413+0000 7fe9cd302700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe9b40a1220 0x7fe9b40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.478 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.413+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe9c00cd9b0 con 0x7fe9c8101200 2023-07-20T19:06:41.478 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.413+0000 7fe9cd302700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe9b40a1220 0x7fe9b40a36e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe9c81006c0 tx=0x7fe9c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:41.478 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.416+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe9c0097610 con 0x7fe9c8101200 2023-07-20T19:06:41.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:41.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/4139589308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:41.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3743304397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:41.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.591 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.592 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:41 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.686 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.627+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe9c802cbc0 con 0x7fe9c8101200 2023-07-20T19:06:41.687 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.627+0000 7fe9be7fc700 1 -- 172.21.15.61:0/1016703890 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe9c009ac30 con 0x7fe9c8101200 2023-07-20T19:06:41.687 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:41.687 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2023-07-20T19:06:41.687 INFO:teuthology.orchestra.run.smithi061.stdout: { 2023-07-20T19:06:41.688 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2023-07-20T19:06:41.688 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": ".mgr" 2023-07-20T19:06:41.688 INFO:teuthology.orchestra.run.smithi061.stdout: } 2023-07-20T19:06:41.688 INFO:teuthology.orchestra.run.smithi061.stdout:] 2023-07-20T19:06:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe9b40a1220 msgr2=0x7fe9b40a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe9b40a1220 0x7fe9b40a36e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe9c81006c0 tx=0x7fe9c4009260 comp rx=0 tx=0).stop 2023-07-20T19:06:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 msgr2=0x7fe9c819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c819dad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0008eb0 tx=0x7fe9c0008ee0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 shutdown_connections 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fe9b40a1220 0x7fe9b40a36e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9c8101200 0x7fe9c819dad0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9c80ff670 0x7fe9c8196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 --2- 172.21.15.61:0/1016703890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c8100860 0x7fe9c8196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 >> 172.21.15.61:0/1016703890 conn(0x7fe9c80faba0 msgr2=0x7fe9c80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 shutdown_connections 2023-07-20T19:06:41.690 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.629+0000 7fe9cf566700 1 -- 172.21.15.61:0/1016703890 wait complete. 2023-07-20T19:06:41.701 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T19:06:41.702 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T19:06:41.835 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 -- 172.21.15.61:0/3554884819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7378102360 msgr2=0x7f7378108c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 --2- 172.21.15.61:0/3554884819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7378102360 0x7f7378108c10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f73700099c0 tx=0x7f7370009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 -- 172.21.15.61:0/3554884819 shutdown_connections 2023-07-20T19:06:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 --2- 172.21.15.61:0/3554884819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7378102360 0x7f7378108c10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 --2- 172.21.15.61:0/3554884819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781007d0 0x7f7378100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 --2- 172.21.15.61:0/3554884819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73781019c0 0x7f7378101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 -- 172.21.15.61:0/3554884819 >> 172.21.15.61:0/3554884819 conn(0x7f73780fbd40 msgr2=0x7f73780fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 -- 172.21.15.61:0/3554884819 shutdown_connections 2023-07-20T19:06:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 -- 172.21.15.61:0/3554884819 wait complete. 2023-07-20T19:06:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.776+0000 7f737d462700 1 Processor -- start 2023-07-20T19:06:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 -- start start 2023-07-20T19:06:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781007d0 0x7f7378074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 0x7f7378072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7378102360 0x7f73780730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378073840 con 0x7f73781019c0 2023-07-20T19:06:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780739b0 con 0x7f7378102360 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73781a03e0 con 0x7f73781007d0 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 0x7f7378072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 0x7f7378072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58428/0 (socket says 172.21.15.61:58428) 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 -- 172.21.15.61:0/3206398052 learned_addr learned my addr 172.21.15.61:0/3206398052 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f7376ffd700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781007d0 0x7f7378074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73777fe700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7378102360 0x7f73780730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.840 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 -- 172.21.15.61:0/3206398052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781007d0 msgr2=0x7f7378074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781007d0 0x7f7378074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 -- 172.21.15.61:0/3206398052 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7378102360 msgr2=0x7f73780730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7378102360 0x7f73780730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 -- 172.21.15.61:0/3206398052 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73700095f0 con 0x7f73781019c0 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73777fe700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7378102360 0x7f73780730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f73767fc700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 0x7f7378072b60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f736000c920 tx=0x7f736000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360007b00 con 0x7f73781019c0 2023-07-20T19:06:41.841 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73781a06c0 con 0x7f73781019c0 2023-07-20T19:06:41.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f736000ce70 con 0x7f73781019c0 2023-07-20T19:06:41.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.777+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360005460 con 0x7f73781019c0 2023-07-20T19:06:41.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.778+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73781a0bf0 con 0x7f73781019c0 2023-07-20T19:06:41.842 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.779+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7378107d10 con 0x7f73781019c0 2023-07-20T19:06:41.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.780+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7360004750 con 0x7f73781019c0 2023-07-20T19:06:41.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.780+0000 7f736ffff700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f73640a1150 0x7f73640a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:41.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.780+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f73600cc080 con 0x7f73781019c0 2023-07-20T19:06:41.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.780+0000 7f7376ffd700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f73640a1150 0x7f73640a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:41.843 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.781+0000 7f7376ffd700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f73640a1150 0x7f73640a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7368009550 tx=0x7f73680091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:41.847 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.787+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7360095c70 con 0x7f73781019c0 2023-07-20T19:06:41.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:41.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/4139589308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:41.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3743304397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:41.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:41.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:41 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/4139589308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3743304397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:41 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.056 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:41.995+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f737802c970 con 0x7f73781019c0 2023-07-20T19:06:42.597 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.537+0000 7f736ffff700 1 -- 172.21.15.61:0/3206398052 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f737802c970 con 0x7f73781019c0 2023-07-20T19:06:42.602 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.541+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f73640a1150 msgr2=0x7f73640a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:42.602 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.541+0000 7f737d462700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f73640a1150 0x7f73640a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7368009550 tx=0x7f73680091b0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.602 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.541+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 msgr2=0x7f7378072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.541+0000 7f737d462700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 0x7f7378072b60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f736000c920 tx=0x7f736000cc30 comp rx=0 tx=0).stop 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.542+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 shutdown_connections 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.542+0000 7f737d462700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f73640a1150 0x7f73640a3610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.542+0000 7f737d462700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7378102360 0x7f73780730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.542+0000 7f737d462700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781007d0 0x7f7378074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.542+0000 7f737d462700 1 --2- 172.21.15.61:0/3206398052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73781019c0 0x7f7378072b60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.542+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 >> 172.21.15.61:0/3206398052 conn(0x7f73780fbd40 msgr2=0x7f7378104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:42.604 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.543+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 shutdown_connections 2023-07-20T19:06:42.604 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.543+0000 7f737d462700 1 -- 172.21.15.61:0/3206398052 wait complete. 2023-07-20T19:06:42.604 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2023-07-20T19:06:42.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T19:06:42.753 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 -- 172.21.15.61:0/2669601808 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840103c00 msgr2=0x7f7840105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:42.754 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 --2- 172.21.15.61:0/2669601808 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840103c00 0x7f7840105ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7834009990 tx=0x7f7834009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.754 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 -- 172.21.15.61:0/2669601808 shutdown_connections 2023-07-20T19:06:42.754 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 --2- 172.21.15.61:0/2669601808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7840106530 0x7f7840108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.754 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 --2- 172.21.15.61:0/2669601808 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f78401036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.754 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 --2- 172.21.15.61:0/2669601808 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840103c00 0x7f7840105ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 -- 172.21.15.61:0/2669601808 >> 172.21.15.61:0/2669601808 conn(0x7f78400fac00 msgr2=0x7f78400fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:42.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 -- 172.21.15.61:0/2669601808 shutdown_connections 2023-07-20T19:06:42.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 -- 172.21.15.61:0/2669601808 wait complete. 2023-07-20T19:06:42.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.694+0000 7f78460c8700 1 Processor -- start 2023-07-20T19:06:42.756 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 -- start start 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f7840196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7840103c00 0x7f78401969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 0x7f784019da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784019dfe0 con 0x7f7840106530 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784019e120 con 0x7f78401012d0 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f78460c8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784019e2b0 con 0x7f7840103c00 2023-07-20T19:06:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f7840196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 0x7f784019da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f7840196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.61:58592/0 (socket says 172.21.15.61:58592) 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 -- 172.21.15.61:0/475540197 learned_addr learned my addr 172.21.15.61:0/475540197 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783ffff700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 0x7f784019da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58444/0 (socket says 172.21.15.61:58444) 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783effd700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7840103c00 0x7f78401969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 -- 172.21.15.61:0/475540197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7840103c00 msgr2=0x7f78401969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7840103c00 0x7f78401969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 -- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 msgr2=0x7f784019da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 0x7f784019da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 -- 172.21.15.61:0/475540197 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830009520 con 0x7f78401012d0 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783ffff700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 0x7f784019da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.695+0000 7f783f7fe700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f7840196460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f783000eb30 tx=0x7f783000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.696+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830009880 con 0x7f78401012d0 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.696+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78300044e0 con 0x7f78401012d0 2023-07-20T19:06:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.696+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78340095f0 con 0x7f78401012d0 2023-07-20T19:06:42.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.696+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830003d80 con 0x7f78401012d0 2023-07-20T19:06:42.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.696+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784019e910 con 0x7f78401012d0 2023-07-20T19:06:42.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.697+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78400fc780 con 0x7f78401012d0 2023-07-20T19:06:42.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.698+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7830003f20 con 0x7f78401012d0 2023-07-20T19:06:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.698+0000 7f783cff9700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f78280a1150 0x7f78280a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.699+0000 7f783effd700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f78280a1150 0x7f78280a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.699+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f7830014070 con 0x7f78401012d0 2023-07-20T19:06:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.699+0000 7f783effd700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f78280a1150 0x7f78280a3610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f783400b2e0 tx=0x7f7834005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.702+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7830096b70 con 0x7f78401012d0 2023-07-20T19:06:42.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:42 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/1016703890' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T19:06:42.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:42 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3206398052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T19:06:42.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:42.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:42.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:42 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:42.968 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:42.909+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f784002ca60 con 0x7f78401012d0 2023-07-20T19:06:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:42 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/1016703890' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T19:06:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:42 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3206398052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T19:06:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:42.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:42 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:42 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/1016703890' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T19:06:43.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:42 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3206398052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T19:06:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:43.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:42 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:43.605 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.545+0000 7f783cff9700 1 -- 172.21.15.61:0/475540197 <== mon.2 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f78300b8e60 con 0x7f78401012d0 2023-07-20T19:06:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.556+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f78280a1150 msgr2=0x7f78280a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.556+0000 7f78460c8700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f78280a1150 0x7f78280a3610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f783400b2e0 tx=0x7f7834005fb0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 msgr2=0x7f7840196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f7840196460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f783000eb30 tx=0x7f783000ee40 comp rx=0 tx=0).stop 2023-07-20T19:06:43.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 shutdown_connections 2023-07-20T19:06:43.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f78280a1150 0x7f78280a3610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7840106530 0x7f784019da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78401012d0 0x7f7840196460 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.618 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 --2- 172.21.15.61:0/475540197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7840103c00 0x7f78401969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.557+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 >> 172.21.15.61:0/475540197 conn(0x7f78400fac00 msgr2=0x7f7840101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.558+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 shutdown_connections 2023-07-20T19:06:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.558+0000 7f78460c8700 1 -- 172.21.15.61:0/475540197 wait complete. 2023-07-20T19:06:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2023-07-20T19:06:43.633 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T19:06:43.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2655236841 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 msgr2=0x7f0bbc10a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:43.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2655236841 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 0x7f0bbc10a920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4009f90 tx=0x7f0bb400b330 comp rx=0 tx=0).stop 2023-07-20T19:06:43.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2655236841 shutdown_connections 2023-07-20T19:06:43.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2655236841 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 0x7f0bbc10a920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2655236841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc074750 0x7f0bbc072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2655236841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc073170 0x7f0bbc0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2655236841 >> 172.21.15.61:0/2655236841 conn(0x7f0bbc0fbb30 msgr2=0x7f0bbc0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:43.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2655236841 shutdown_connections 2023-07-20T19:06:43.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.723+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2655236841 wait complete. 2023-07-20T19:06:43.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 Processor -- start 2023-07-20T19:06:43.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 -- start start 2023-07-20T19:06:43.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc073170 0x7f0bbc198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:43.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 0x7f0bbc1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:43.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 0x7f0bbc19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:43.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1a0010 con 0x7f0bbc074750 2023-07-20T19:06:43.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1a0150 con 0x7f0bbc1084b0 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.724+0000 7f0bc0fe3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1a02e0 con 0x7f0bbc073170 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 0x7f0bbc1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 0x7f0bbc1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58456/0 (socket says 172.21.15.61:58456) 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 -- 172.21.15.61:0/2511380560 learned_addr learned my addr 172.21.15.61:0/2511380560 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbb7fe700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc073170 0x7f0bbc198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbbfff700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 0x7f0bbc19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 -- 172.21.15.61:0/2511380560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc073170 msgr2=0x7f0bbc198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc073170 0x7f0bbc198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 -- 172.21.15.61:0/2511380560 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 msgr2=0x7f0bbc19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 0x7f0bbc19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 -- 172.21.15.61:0/2511380560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00095f0 con 0x7f0bbc074750 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbb7fe700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc073170 0x7f0bbc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bbaffd700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 0x7f0bbc1989d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000ec10 tx=0x7f0bb000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000cc90 con 0x7f0bbc074750 2023-07-20T19:06:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb4009c40 con 0x7f0bbc074750 2023-07-20T19:06:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb00044e0 con 0x7f0bbc074750 2023-07-20T19:06:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0010430 con 0x7f0bbc074750 2023-07-20T19:06:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.725+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc1a0940 con 0x7f0bbc074750 2023-07-20T19:06:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.726+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc106bb0 con 0x7f0bbc074750 2023-07-20T19:06:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.728+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0bb0003f90 con 0x7f0bbc074750 2023-07-20T19:06:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.728+0000 7f0bb8ff9700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0ba40a11f0 0x7f0ba40a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.729+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f0bb0014070 con 0x7f0bbc074750 2023-07-20T19:06:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.729+0000 7f0bbb7fe700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0ba40a11f0 0x7f0ba40a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.729+0000 7f0bbb7fe700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0ba40a11f0 0x7f0ba40a36b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0bac011490 tx=0x7f0bac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.732+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0bb0096700 con 0x7f0bbc074750 2023-07-20T19:06:43.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3206398052' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T19:06:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T19:06:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/475540197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T19:06:43.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:43 smithi156 ceph-mon[129228]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T19:06:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3206398052' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T19:06:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T19:06:43.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:43.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:43.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/475540197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T19:06:43.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:43 smithi061 ceph-mon[127454]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T19:06:44.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:44.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3206398052' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T19:06:44.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T19:06:44.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T19:06:44.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:44.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/475540197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T19:06:44.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:43 smithi149 ceph-mon[128885]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T19:06:44.058 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:43.998+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0bbc02cbe0 con 0x7f0bbc074750 2023-07-20T19:06:44.632 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.572+0000 7f0bb8ff9700 1 -- 172.21.15.61:0/2511380560 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0bb0099d20 con 0x7f0bbc074750 2023-07-20T19:06:44.644 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.581+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0ba40a11f0 msgr2=0x7f0ba40a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:44.644 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.581+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0ba40a11f0 0x7f0ba40a36b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0bac011490 tx=0x7f0bac009260 comp rx=0 tx=0).stop 2023-07-20T19:06:44.644 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.581+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 msgr2=0x7f0bbc1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:44.644 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.581+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 0x7f0bbc1989d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000ec10 tx=0x7f0bb000ef20 comp rx=0 tx=0).stop 2023-07-20T19:06:44.651 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 shutdown_connections 2023-07-20T19:06:44.651 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f0ba40a11f0 0x7f0ba40a36b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bbc1084b0 0x7f0bbc19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc073170 0x7f0bbc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 --2- 172.21.15.61:0/2511380560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bbc074750 0x7f0bbc1989d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 >> 172.21.15.61:0/2511380560 conn(0x7f0bbc0fbb30 msgr2=0x7f0bbc10a640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 shutdown_connections 2023-07-20T19:06:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.592+0000 7f0bc0fe3700 1 -- 172.21.15.61:0/2511380560 wait complete. 2023-07-20T19:06:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T19:06:44.664 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T19:06:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 -- 172.21.15.61:0/57112874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf0101270 msgr2=0x7f1bf0103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 --2- 172.21.15.61:0/57112874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf0101270 0x7f1bf0103660 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009990 tx=0x7f1be0009ca0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 -- 172.21.15.61:0/57112874 shutdown_connections 2023-07-20T19:06:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 --2- 172.21.15.61:0/57112874 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1bf01064d0 0x7f1bf01088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 --2- 172.21.15.61:0/57112874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf0101270 0x7f1bf0103660 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 --2- 172.21.15.61:0/57112874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0103ba0 0x7f1bf0105f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 -- 172.21.15.61:0/57112874 >> 172.21.15.61:0/57112874 conn(0x7f1bf00faba0 msgr2=0x7f1bf00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 -- 172.21.15.61:0/57112874 shutdown_connections 2023-07-20T19:06:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.746+0000 7f1bf6327700 1 -- 172.21.15.61:0/57112874 wait complete. 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 Processor -- start 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 -- start start 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0101270 0x7f1bf0196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1bf0103ba0 0x7f1bf0196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 0x7f1bf019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf019dfb0 con 0x7f1bf01064d0 2023-07-20T19:06:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf019e0f0 con 0x7f1bf0103ba0 2023-07-20T19:06:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf6327700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf019e280 con 0x7f1bf0101270 2023-07-20T19:06:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 0x7f1bf019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 0x7f1bf019d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58480/0 (socket says 172.21.15.61:58480) 2023-07-20T19:06:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 -- 172.21.15.61:0/3779155554 learned_addr learned my addr 172.21.15.61:0/3779155554 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bef7fe700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1bf0103ba0 0x7f1bf0196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1beffff700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0101270 0x7f1bf0196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 -- 172.21.15.61:0/3779155554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0101270 msgr2=0x7f1bf0196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0101270 0x7f1bf0196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 -- 172.21.15.61:0/3779155554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1bf0103ba0 msgr2=0x7f1bf0196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1bf0103ba0 0x7f1bf0196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1bf48c4700 1 -- 172.21.15.61:0/3779155554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be00095f0 con 0x7f1bf01064d0 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.747+0000 7f1beffff700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0101270 0x7f1bf0196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:06:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.748+0000 7f1bf48c4700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 0x7f1bf019d9e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007ee0 tx=0x7f1be800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.748+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be800ddd0 con 0x7f1bf01064d0 2023-07-20T19:06:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.748+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf019e560 con 0x7f1bf01064d0 2023-07-20T19:06:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.748+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1be800f040 con 0x7f1bf01064d0 2023-07-20T19:06:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.748+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be80048e0 con 0x7f1bf01064d0 2023-07-20T19:06:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.748+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf019ea70 con 0x7f1bf01064d0 2023-07-20T19:06:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.749+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf00fc720 con 0x7f1bf01064d0 2023-07-20T19:06:44.813 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.751+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1be8003a00 con 0x7f1bf01064d0 2023-07-20T19:06:44.813 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.751+0000 7f1bed7fa700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1bd80a11a0 0x7f1bd80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:44.813 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.752+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1be80cc9a0 con 0x7f1bf01064d0 2023-07-20T19:06:44.813 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.752+0000 7f1beffff700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1bd80a11a0 0x7f1bd80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:44.814 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.754+0000 7f1beffff700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1bd80a11a0 0x7f1bd80a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1be0006010 tx=0x7f1be000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:44.815 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.755+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1be8096240 con 0x7f1bf01064d0 2023-07-20T19:06:44.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:44 smithi156 ceph-mon[129228]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T19:06:44.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:44 smithi156 ceph-mon[129228]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T19:06:44.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:44 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2511380560' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T19:06:44.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:44 smithi156 ceph-mon[129228]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T19:06:44.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:44 smithi061 ceph-mon[127454]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T19:06:44.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:44 smithi061 ceph-mon[127454]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T19:06:44.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:44 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2511380560' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T19:06:44.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:44 smithi061 ceph-mon[127454]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T19:06:45.039 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.979+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf002cb70 con 0x7f1bf01064d0 2023-07-20T19:06:45.040 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.981+0000 7f1bed7fa700 1 -- 172.21.15.61:0/3779155554 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22118 (secure 0 0 0) 0x7f1be8023090 con 0x7f1bf01064d0 2023-07-20T19:06:45.041 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:45.041 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","created":"2023-07-20T18:59:47.496551+0000","modified":"2023-07-20T19:06:44.000508+0000","last_up_change":"2023-07-20T19:06:03.755886+0000","last_in_change":"2023-07-20T19:05:46.242910+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T19:02:56.142587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T19:06:42.431216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T19:06:43.357852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfae06af-18f4-487e-ae4b-8e8c9d36ef76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6803","nonce":1072462099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6805","nonce":1072462099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6809","nonce":1072462099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6807","nonce":1072462099}]},"public_addr":"172.21.15.61:6803/1072462099","cluster_addr":"172.21.15.61:6805/1072462099","heartbeat_back_addr":"172.21.15.61:6809/1072462099","heartbeat_front_addr":"172.21.15.61:6807/1072462099","state":["exists","up"]},{"osd":1,"uuid":"d3e2b1d4-abe5-4438-9196-6ef26230a071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6801","nonce":414280084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6803","nonce":414280084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6807","nonce":414280084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6805","nonce":414280084}]},"public_addr":"172.21.15.149:6801/414280084","cluster_addr":"172.21.15.149:6803/414280084","heartbeat_back_addr":"172.21.15.149:6807/414280084","heartbeat_front_addr":"172.21.15.149:6805/414280084","state":["exists","up"]},{"osd":2,"uuid":"34a03029-87a9-49e1-b262-32be5070df0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6801","nonce":738107812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6803","nonce":738107812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6807","nonce":738107812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6805","nonce":738107812}]},"public_addr":"172.21.15.156:6801/738107812","cluster_addr":"172.21.15.156:6803/738107812","heartbeat_back_addr":"172.21.15.156:6807/738107812","heartbeat_front_addr":"172.21.15.156:6805/738107812","state":["exists","up"]},{"osd":3,"uuid":"c5711133-530f-488a-9db4-2b8a61b7ea02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6811","nonce":1371683823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6813","nonce":1371683823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6817","nonce":1371683823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6815","nonce":1371683823}]},"public_addr":"172.21.15.61:6811/1371683823","cluster_addr":"172.21.15.61:6813/1371683823","heartbeat_back_addr":"172.21.15.61:6817/1371683823","heartbeat_front_addr":"172.21.15.61:6815/1371683823","state":["exists","up"]},{"osd":4,"uuid":"6062cdd3-5bf1-458a-8454-253627db736f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6809","nonce":3513915816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6811","nonce":3513915816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6815","nonce":3513915816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6813","nonce":3513915816}]},"public_addr":"172.21.15.149:6809/3513915816","cluster_addr":"172.21.15.149:6811/3513915816","heartbeat_back_addr":"172.21.15.149:6815/3513915816","heartbeat_front_addr":"172.21.15.149:6813/3513915816","state":["exists","up"]},{"osd":5,"uuid":"1fae1b15-43cb-40d9-ad90-73af4cc96197","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6809","nonce":211953844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6811","nonce":211953844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6815","nonce":211953844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6813","nonce":211953844}]},"public_addr":"172.21.15.156:6809/211953844","cluster_addr":"172.21.15.156:6811/211953844","heartbeat_back_addr":"172.21.15.156:6815/211953844","heartbeat_front_addr":"172.21.15.156:6813/211953844","state":["exists","up"]},{"osd":6,"uuid":"84316d15-3a51-432c-9d02-57270ac09769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6819","nonce":1770665297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6821","nonce":1770665297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6825","nonce":1770665297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6823","nonce":1770665297}]},"public_addr":"172.21.15.61:6819/1770665297","cluster_addr":"172.21.15.61:6821/1770665297","heartbeat_back_addr":"172.21.15.61:6825/1770665297","heartbeat_front_addr":"172.21.15.61:6823/1770665297","state":["exists","up"]},{"osd":7,"uuid":"705c1527-9342-402c-add4-4382e005dbfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6817","nonce":1095358817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6819","nonce":1095358817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6823","nonce":1095358817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6821","nonce":1095358817}]},"public_addr":"172.21.15.149:6817/1095358817","cluster_addr":"172.21.15.149:6819/1095358817","heartbeat_back_addr":"172.21.15.149:6823/1095358817","heartbeat_front_addr":"172.21.15.149:6821/1095358817","state":["exists","up"]},{"osd":8,"uuid":"6e790866-ee13-4c27-bbd4-ea90e853de1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6817","nonce":1602221626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6819","nonce":1602221626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6823","nonce":1602221626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6821","nonce":1602221626}]},"public_addr":"172.21.15.156:6817/1602221626","cluster_addr":"172.21.15.156:6819/1602221626","heartbeat_back_addr":"172.21.15.156:6823/1602221626","heartbeat_front_addr":"172.21.15.156:6821/1602221626","state":["exists","up"]},{"osd":9,"uuid":"ac0b69d5-ccea-470d-a8c8-b9a2f508af24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6827","nonce":2654664682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6829","nonce":2654664682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6833","nonce":2654664682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6831","nonce":2654664682}]},"public_addr":"172.21.15.61:6827/2654664682","cluster_addr":"172.21.15.61:6829/2654664682","heartbeat_back_addr":"172.21.15.61:6833/2654664682","heartbeat_front_addr":"172.21.15.61:6831/2654664682","state":["exists","up"]},{"osd":10,"uuid":"1a6e7d98-a3f2-4d81-843e-bca056ec748c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6825","nonce":1912645375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6827","nonce":1912645375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6831","nonce":1912645375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6829","nonce":1912645375}]},"public_addr":"172.21.15.149:6825/1912645375","cluster_addr":"172.21.15.149:6827/1912645375","heartbeat_back_addr":"172.21.15.149:6831/1912645375","heartbeat_front_addr":"172.21.15.149:6829/1912645375","state":["exists","up"]},{"osd":11,"uuid":"5bcb26b2-0789-4cf8-9803-98a6da33d5d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6825","nonce":893354887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6827","nonce":893354887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6831","nonce":893354887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6829","nonce":893354887}]},"public_addr":"172.21.15.156:6825/893354887","cluster_addr":"172.21.15.156:6827/893354887","heartbeat_back_addr":"172.21.15.156:6831/893354887","heartbeat_front_addr":"172.21.15.156:6829/893354887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:09.841610+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:31.306806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:52.704303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:13.230276+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:34.832690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:55.534516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:16.097468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:37.618136+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:58.685044+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:19.824839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:41.039703+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:06:02.038214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.61:6801/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6800/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/3344679989":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6801/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/170145058":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/575168883":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/2333936767":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/74416212":"2023-07-21T19:00:21.461577+0000","172.21.15.61:6800/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/2426181232":"2023-07-21T19:00:21.461577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T19:06:45.042 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1bd80a11a0 msgr2=0x7f1bd80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.043 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1bd80a11a0 0x7f1bd80a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1be0006010 tx=0x7f1be000b540 comp rx=0 tx=0).stop 2023-07-20T19:06:45.043 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 msgr2=0x7f1bf019d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.043 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 0x7f1bf019d9e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007ee0 tx=0x7f1be800d3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.043 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 shutdown_connections 2023-07-20T19:06:45.043 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f1bd80a11a0 0x7f1bd80a3660 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.043 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf01064d0 0x7f1bf019d9e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bf0101270 0x7f1bf0196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 --2- 172.21.15.61:0/3779155554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1bf0103ba0 0x7f1bf0196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 >> 172.21.15.61:0/3779155554 conn(0x7f1bf00faba0 msgr2=0x7f1bf0101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:45.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 shutdown_connections 2023-07-20T19:06:45.044 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:44.983+0000 7f1bf6327700 1 -- 172.21.15.61:0/3779155554 wait complete. 2023-07-20T19:06:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:44 smithi149 ceph-mon[128885]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T19:06:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:44 smithi149 ceph-mon[128885]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T19:06:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:44 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2511380560' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T19:06:45.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:44 smithi149 ceph-mon[128885]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-07-20T19:06:45.056 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T19:06:45.187 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.127+0000 7f3425450700 1 -- 172.21.15.61:0/1001837090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34200ff6a0 msgr2=0x7f34200ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.127+0000 7f3425450700 1 --2- 172.21.15.61:0/1001837090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34200ff6a0 0x7f34200ffac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34100099c0 tx=0x7f3410009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.127+0000 7f3425450700 1 -- 172.21.15.61:0/1001837090 shutdown_connections 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.127+0000 7f3425450700 1 --2- 172.21.15.61:0/1001837090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3420101230 0x7f3420107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.127+0000 7f3425450700 1 --2- 172.21.15.61:0/1001837090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34200ff6a0 0x7f34200ffac0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.127+0000 7f3425450700 1 --2- 172.21.15.61:0/1001837090 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3420100890 0x7f3420100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 -- 172.21.15.61:0/1001837090 >> 172.21.15.61:0/1001837090 conn(0x7f34200fabf0 msgr2=0x7f34200fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 -- 172.21.15.61:0/1001837090 shutdown_connections 2023-07-20T19:06:45.189 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 -- 172.21.15.61:0/1001837090 wait complete. 2023-07-20T19:06:45.190 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 Processor -- start 2023-07-20T19:06:45.190 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 -- start start 2023-07-20T19:06:45.190 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34200ff6a0 0x7f34200696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.128+0000 7f3425450700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3420100890 0x7f342019b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3425450700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 0x7f342019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3425450700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342019df40 con 0x7f3420101230 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3425450700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342019e0b0 con 0x7f3420100890 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3425450700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342019e270 con 0x7f34200ff6a0 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 0x7f342019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 0x7f342019d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58494/0 (socket says 172.21.15.61:58494) 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 -- 172.21.15.61:0/2829536097 learned_addr learned my addr 172.21.15.61:0/2829536097 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341e7fc700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3420100890 0x7f342019b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341effd700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34200ff6a0 0x7f34200696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 -- 172.21.15.61:0/2829536097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34200ff6a0 msgr2=0x7f34200696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34200ff6a0 0x7f34200696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 -- 172.21.15.61:0/2829536097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3420100890 msgr2=0x7f342019b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3420100890 0x7f342019b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 -- 172.21.15.61:0/2829536097 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100095f0 con 0x7f3420101230 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f341f7fe700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 0x7f342019d770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3418007ee0 tx=0x7f341800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418017070 con 0x7f3420101230 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f341800f040 con 0x7f3420101230 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342019e550 con 0x7f3420101230 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418005580 con 0x7f3420101230 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.129+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342019ea10 con 0x7f3420101230 2023-07-20T19:06:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.130+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3420106b90 con 0x7f3420101230 2023-07-20T19:06:45.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.131+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f341800db20 con 0x7f3420101230 2023-07-20T19:06:45.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.132+0000 7f3407fff700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f34080a11a0 0x7f34080a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.132+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f34180cbef0 con 0x7f3420101230 2023-07-20T19:06:45.195 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.132+0000 7f341effd700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f34080a11a0 0x7f34080a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.195 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.132+0000 7f341effd700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f34080a11a0 0x7f34080a3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3410009f50 tx=0x7f341000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:45.196 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.136+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3418095790 con 0x7f3420101230 2023-07-20T19:06:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.351+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f342002c970 con 0x7f3420101230 2023-07-20T19:06:45.412 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.353+0000 7f3407fff700 1 -- 172.21.15.61:0/2829536097 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22118 (secure 0 0 0) 0x7f3418098db0 con 0x7f3420101230 2023-07-20T19:06:45.413 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-07-20T19:06:45.413 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"fsid":"7541eb30-272f-11ee-9b34-001a4aab830c","created":"2023-07-20T18:59:47.496551+0000","modified":"2023-07-20T19:06:44.000508+0000","last_up_change":"2023-07-20T19:06:03.755886+0000","last_in_change":"2023-07-20T19:05:46.242910+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T19:02:56.142587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T19:06:42.431216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T19:06:43.357852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfae06af-18f4-487e-ae4b-8e8c9d36ef76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6803","nonce":1072462099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6805","nonce":1072462099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6809","nonce":1072462099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":1072462099},{"type":"v1","addr":"172.21.15.61:6807","nonce":1072462099}]},"public_addr":"172.21.15.61:6803/1072462099","cluster_addr":"172.21.15.61:6805/1072462099","heartbeat_back_addr":"172.21.15.61:6809/1072462099","heartbeat_front_addr":"172.21.15.61:6807/1072462099","state":["exists","up"]},{"osd":1,"uuid":"d3e2b1d4-abe5-4438-9196-6ef26230a071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6801","nonce":414280084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6803","nonce":414280084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6807","nonce":414280084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":414280084},{"type":"v1","addr":"172.21.15.149:6805","nonce":414280084}]},"public_addr":"172.21.15.149:6801/414280084","cluster_addr":"172.21.15.149:6803/414280084","heartbeat_back_addr":"172.21.15.149:6807/414280084","heartbeat_front_addr":"172.21.15.149:6805/414280084","state":["exists","up"]},{"osd":2,"uuid":"34a03029-87a9-49e1-b262-32be5070df0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6801","nonce":738107812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6803","nonce":738107812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6807","nonce":738107812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":738107812},{"type":"v1","addr":"172.21.15.156:6805","nonce":738107812}]},"public_addr":"172.21.15.156:6801/738107812","cluster_addr":"172.21.15.156:6803/738107812","heartbeat_back_addr":"172.21.15.156:6807/738107812","heartbeat_front_addr":"172.21.15.156:6805/738107812","state":["exists","up"]},{"osd":3,"uuid":"c5711133-530f-488a-9db4-2b8a61b7ea02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6811","nonce":1371683823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6813","nonce":1371683823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6817","nonce":1371683823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":1371683823},{"type":"v1","addr":"172.21.15.61:6815","nonce":1371683823}]},"public_addr":"172.21.15.61:6811/1371683823","cluster_addr":"172.21.15.61:6813/1371683823","heartbeat_back_addr":"172.21.15.61:6817/1371683823","heartbeat_front_addr":"172.21.15.61:6815/1371683823","state":["exists","up"]},{"osd":4,"uuid":"6062cdd3-5bf1-458a-8454-253627db736f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6809","nonce":3513915816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6811","nonce":3513915816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6815","nonce":3513915816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":3513915816},{"type":"v1","addr":"172.21.15.149:6813","nonce":3513915816}]},"public_addr":"172.21.15.149:6809/3513915816","cluster_addr":"172.21.15.149:6811/3513915816","heartbeat_back_addr":"172.21.15.149:6815/3513915816","heartbeat_front_addr":"172.21.15.149:6813/3513915816","state":["exists","up"]},{"osd":5,"uuid":"1fae1b15-43cb-40d9-ad90-73af4cc96197","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6809","nonce":211953844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6811","nonce":211953844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6815","nonce":211953844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":211953844},{"type":"v1","addr":"172.21.15.156:6813","nonce":211953844}]},"public_addr":"172.21.15.156:6809/211953844","cluster_addr":"172.21.15.156:6811/211953844","heartbeat_back_addr":"172.21.15.156:6815/211953844","heartbeat_front_addr":"172.21.15.156:6813/211953844","state":["exists","up"]},{"osd":6,"uuid":"84316d15-3a51-432c-9d02-57270ac09769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6819","nonce":1770665297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6821","nonce":1770665297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6825","nonce":1770665297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":1770665297},{"type":"v1","addr":"172.21.15.61:6823","nonce":1770665297}]},"public_addr":"172.21.15.61:6819/1770665297","cluster_addr":"172.21.15.61:6821/1770665297","heartbeat_back_addr":"172.21.15.61:6825/1770665297","heartbeat_front_addr":"172.21.15.61:6823/1770665297","state":["exists","up"]},{"osd":7,"uuid":"705c1527-9342-402c-add4-4382e005dbfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6817","nonce":1095358817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6819","nonce":1095358817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6823","nonce":1095358817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":1095358817},{"type":"v1","addr":"172.21.15.149:6821","nonce":1095358817}]},"public_addr":"172.21.15.149:6817/1095358817","cluster_addr":"172.21.15.149:6819/1095358817","heartbeat_back_addr":"172.21.15.149:6823/1095358817","heartbeat_front_addr":"172.21.15.149:6821/1095358817","state":["exists","up"]},{"osd":8,"uuid":"6e790866-ee13-4c27-bbd4-ea90e853de1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6817","nonce":1602221626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6819","nonce":1602221626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6823","nonce":1602221626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1602221626},{"type":"v1","addr":"172.21.15.156:6821","nonce":1602221626}]},"public_addr":"172.21.15.156:6817/1602221626","cluster_addr":"172.21.15.156:6819/1602221626","heartbeat_back_addr":"172.21.15.156:6823/1602221626","heartbeat_front_addr":"172.21.15.156:6821/1602221626","state":["exists","up"]},{"osd":9,"uuid":"ac0b69d5-ccea-470d-a8c8-b9a2f508af24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6827","nonce":2654664682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6829","nonce":2654664682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6833","nonce":2654664682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":2654664682},{"type":"v1","addr":"172.21.15.61:6831","nonce":2654664682}]},"public_addr":"172.21.15.61:6827/2654664682","cluster_addr":"172.21.15.61:6829/2654664682","heartbeat_back_addr":"172.21.15.61:6833/2654664682","heartbeat_front_addr":"172.21.15.61:6831/2654664682","state":["exists","up"]},{"osd":10,"uuid":"1a6e7d98-a3f2-4d81-843e-bca056ec748c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6825","nonce":1912645375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6827","nonce":1912645375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6831","nonce":1912645375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":1912645375},{"type":"v1","addr":"172.21.15.149:6829","nonce":1912645375}]},"public_addr":"172.21.15.149:6825/1912645375","cluster_addr":"172.21.15.149:6827/1912645375","heartbeat_back_addr":"172.21.15.149:6831/1912645375","heartbeat_front_addr":"172.21.15.149:6829/1912645375","state":["exists","up"]},{"osd":11,"uuid":"5bcb26b2-0789-4cf8-9803-98a6da33d5d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6825","nonce":893354887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6827","nonce":893354887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6831","nonce":893354887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":893354887},{"type":"v1","addr":"172.21.15.156:6829","nonce":893354887}]},"public_addr":"172.21.15.156:6825/893354887","cluster_addr":"172.21.15.156:6827/893354887","heartbeat_back_addr":"172.21.15.156:6831/893354887","heartbeat_front_addr":"172.21.15.156:6829/893354887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:09.841610+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:31.306806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:02:52.704303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:13.230276+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:34.832690+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:03:55.534516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:16.097468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:37.618136+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:04:58.685044+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:19.824839+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:05:41.039703+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T19:06:02.038214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.61:6801/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6800/2098378944":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/3344679989":"2023-07-21T19:00:47.602758+0000","172.21.15.61:6801/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/170145058":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/575168883":"2023-07-21T19:00:47.602758+0000","172.21.15.61:0/2333936767":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/74416212":"2023-07-21T19:00:21.461577+0000","172.21.15.61:6800/2016235088":"2023-07-21T19:00:21.461577+0000","172.21.15.61:0/2426181232":"2023-07-21T19:00:21.461577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T19:06:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f34080a11a0 msgr2=0x7f34080a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f34080a11a0 0x7f34080a3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3410009f50 tx=0x7f341000b540 comp rx=0 tx=0).stop 2023-07-20T19:06:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 msgr2=0x7f342019d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 0x7f342019d770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3418007ee0 tx=0x7f341800d3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 shutdown_connections 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f34080a11a0 0x7f34080a3660 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3420101230 0x7f342019d770 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34200ff6a0 0x7f34200696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 --2- 172.21.15.61:0/2829536097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3420100890 0x7f342019b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.355+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 >> 172.21.15.61:0/2829536097 conn(0x7f34200fabf0 msgr2=0x7f3420103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.356+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 shutdown_connections 2023-07-20T19:06:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.356+0000 7f3425450700 1 -- 172.21.15.61:0/2829536097 wait complete. 2023-07-20T19:06:45.429 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T19:06:45.560 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.500+0000 7f31dcb4d700 1 -- 172.21.15.61:0/1334869374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d81064a0 msgr2=0x7f31d81088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.500+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/1334869374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d81064a0 0x7f31d81088b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f31d000b0a0 tx=0x7f31d000b3b0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 -- 172.21.15.61:0/1334869374 shutdown_connections 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/1334869374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d81064a0 0x7f31d81088b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/1334869374 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d8101240 0x7f31d8103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/1334869374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d8103b70 0x7f31d8105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 -- 172.21.15.61:0/1334869374 >> 172.21.15.61:0/1334869374 conn(0x7f31d80fabd0 msgr2=0x7f31d80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:45.561 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 -- 172.21.15.61:0/1334869374 shutdown_connections 2023-07-20T19:06:45.562 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 -- 172.21.15.61:0/1334869374 wait complete. 2023-07-20T19:06:45.563 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 Processor -- start 2023-07-20T19:06:45.563 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.501+0000 7f31dcb4d700 1 -- start start 2023-07-20T19:06:45.563 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d8101240 0x7f31d81987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 0x7f31d8198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d81064a0 0x7f31d819fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d81a0340 con 0x7f31d8103b70 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d81a0480 con 0x7f31d81064a0 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d81a0610 con 0x7f31d8101240 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 0x7f31d8198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 0x7f31d8198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58506/0 (socket says 172.21.15.61:58506) 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 -- 172.21.15.61:0/253174943 learned_addr learned my addr 172.21.15.61:0/253174943 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d659c700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d8101240 0x7f31d81987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d6d9d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d81064a0 0x7f31d819fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 -- 172.21.15.61:0/253174943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d8101240 msgr2=0x7f31d81987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d8101240 0x7f31d81987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 -- 172.21.15.61:0/253174943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d81064a0 msgr2=0x7f31d819fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d81064a0 0x7f31d819fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 -- 172.21.15.61:0/253174943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0009d00 con 0x7f31d8103b70 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d6d9d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d81064a0 0x7f31d819fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31d5d9b700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 0x7f31d8198d00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00d950 tx=0x7f31cc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00f890 con 0x7f31d8103b70 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31cc010460 con 0x7f31d8103b70 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d81a08f0 con 0x7f31d8103b70 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00e610 con 0x7f31d8103b70 2023-07-20T19:06:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.502+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d81a0e00 con 0x7f31d8103b70 2023-07-20T19:06:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.503+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d8108020 con 0x7f31d8103b70 2023-07-20T19:06:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.505+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f31cc0108d0 con 0x7f31d8103b70 2023-07-20T19:06:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.505+0000 7f31c77fe700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f31c00a1150 0x7f31c00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.505+0000 7f31d659c700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f31c00a1150 0x7f31c00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:45.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.505+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f31cc0cc580 con 0x7f31d8103b70 2023-07-20T19:06:45.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.506+0000 7f31d659c700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f31c00a1150 0x7f31c00a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009c60 tx=0x7f31c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:45.568 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.509+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f31cc095e20 con 0x7f31d8103b70 2023-07-20T19:06:45.720 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: pgmap v247: 97 pgs: 9 creating+peering, 1 active+clean, 87 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:45.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T19:06:45.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2511380560' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T19:06:45.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T19:06:45.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T19:06:45.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3779155554' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:45.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.723 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2829536097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:45.723 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.723 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.723 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.724 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.725 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:45.725 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:45.725 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:45 smithi061 ceph-mon[127454]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.821 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:45.761+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f31d802c970 con 0x7f31d8103b70 2023-07-20T19:06:45.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: pgmap v247: 97 pgs: 9 creating+peering, 1 active+clean, 87 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:45.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T19:06:45.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2511380560' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T19:06:45.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T19:06:45.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T19:06:45.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3779155554' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:45.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2829536097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:45.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:45.920 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:45.920 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:45 smithi156 ceph-mon[129228]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: pgmap v247: 97 pgs: 9 creating+peering, 1 active+clean, 87 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2511380560' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3779155554' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2829536097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T19:06:46.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T19:06:46.057 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:45 smithi149 ceph-mon[128885]: from='mgr.14152 172.21.15.61:0/543763512' entity='mgr.x' 2023-07-20T19:06:46.637 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.577+0000 7f31c77fe700 1 -- 172.21.15.61:0/253174943 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f31cc0160b0 con 0x7f31d8103b70 2023-07-20T19:06:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f31c00a1150 msgr2=0x7f31c00a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f31c00a1150 0x7f31c00a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009c60 tx=0x7f31c8009260 comp rx=0 tx=0).stop 2023-07-20T19:06:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 msgr2=0x7f31d8198d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 0x7f31d8198d00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00d950 tx=0x7f31cc00dc60 comp rx=0 tx=0).stop 2023-07-20T19:06:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 shutdown_connections 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f31c00a1150 0x7f31c00a3610 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31d81064a0 0x7f31d819fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d8101240 0x7f31d81987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 --2- 172.21.15.61:0/253174943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31d8103b70 0x7f31d8198d00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 >> 172.21.15.61:0/253174943 conn(0x7f31d80fabd0 msgr2=0x7f31d80ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 shutdown_connections 2023-07-20T19:06:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.580+0000 7f31dcb4d700 1 -- 172.21.15.61:0/253174943 wait complete. 2023-07-20T19:06:46.654 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-07-20T19:06:46.654 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-20T19:06:46.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 -- 172.21.15.61:0/2928352723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2341022f0 msgr2=0x7fa234108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:46.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/2928352723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2341022f0 0x7fa234108ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00b0d0 tx=0x7fa22c00b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 -- 172.21.15.61:0/2928352723 shutdown_connections 2023-07-20T19:06:46.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/2928352723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2341022f0 0x7fa234108ba0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/2928352723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa234100760 0x7fa234100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/2928352723 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa234101950 0x7fa234101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 -- 172.21.15.61:0/2928352723 >> 172.21.15.61:0/2928352723 conn(0x7fa2340fbcf0 msgr2=0x7fa2340fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:46.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 -- 172.21.15.61:0/2928352723 shutdown_connections 2023-07-20T19:06:46.787 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.726+0000 7fa23ab4f700 1 -- 172.21.15.61:0/2928352723 wait complete. 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 Processor -- start 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 -- start start 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 0x7fa234074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa234101950 0x7fa234072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2341022f0 0x7fa2340730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234073840 con 0x7fa234100760 2023-07-20T19:06:46.789 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2340739b0 con 0x7fa2341022f0 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa23ab4f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341a05e0 con 0x7fa234101950 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 0x7fa234074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 0x7fa234074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58518/0 (socket says 172.21.15.61:58518) 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 -- 172.21.15.61:0/3165134559 learned_addr learned my addr 172.21.15.61:0/3165134559 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa233fff700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa234101950 0x7fa234072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2390ec700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2341022f0 0x7fa2340730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:46.790 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 -- 172.21.15.61:0/3165134559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa234101950 msgr2=0x7fa234072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa234101950 0x7fa234072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 -- 172.21.15.61:0/3165134559 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2341022f0 msgr2=0x7fa2340730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2341022f0 0x7fa2340730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 -- 172.21.15.61:0/3165134559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c009d00 con 0x7fa234100760 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa2388eb700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 0x7fa234074520 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa22800d980 tx=0x7fa22800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.727+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22800f890 con 0x7fa234100760 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.728+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa228010460 con 0x7fa234100760 2023-07-20T19:06:46.791 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.728+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2280195c0 con 0x7fa234100760 2023-07-20T19:06:46.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.728+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2341a08c0 con 0x7fa234100760 2023-07-20T19:06:46.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.728+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2341a0d80 con 0x7fa234100760 2023-07-20T19:06:46.792 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.729+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234107cc0 con 0x7fa234100760 2023-07-20T19:06:46.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.730+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa2280105d0 con 0x7fa234100760 2023-07-20T19:06:46.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.730+0000 7fa231ffb700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1150 0x7fa2240a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:46.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.731+0000 7fa233fff700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1150 0x7fa2240a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:46.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.731+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa2280cc520 con 0x7fa234100760 2023-07-20T19:06:46.793 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.731+0000 7fa233fff700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1150 0x7fa2240a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa220005950 tx=0x7fa22000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:46.794 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.735+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa228095dc0 con 0x7fa234100760 2023-07-20T19:06:46.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:46 smithi156 ceph-mon[129228]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T19:06:46.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:46 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6838/1971202302,v1:172.21.15.156:6839/1971202302] up:active 2023-07-20T19:06:46.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:46 smithi156 ceph-mon[129228]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:46.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:46 smithi156 ceph-mon[129228]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:46.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:46 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/253174943' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T19:06:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:46 smithi061 ceph-mon[127454]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T19:06:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:46 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6838/1971202302,v1:172.21.15.156:6839/1971202302] up:active 2023-07-20T19:06:46.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:46 smithi061 ceph-mon[127454]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:46.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:46 smithi061 ceph-mon[127454]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:46.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:46 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/253174943' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T19:06:47.046 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:46.986+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fa23402cbe0 con 0x7fa234100760 2023-07-20T19:06:47.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:46 smithi149 ceph-mon[128885]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T19:06:47.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:46 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6838/1971202302,v1:172.21.15.156:6839/1971202302] up:active 2023-07-20T19:06:47.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:46 smithi149 ceph-mon[128885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:47.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:46 smithi149 ceph-mon[128885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:47.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:46 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/253174943' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T19:06:47.584 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.524+0000 7fa231ffb700 1 -- 172.21.15.61:0/3165134559 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa2280993e0 con 0x7fa234100760 2023-07-20T19:06:47.587 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1150 msgr2=0x7fa2240a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:47.587 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1150 0x7fa2240a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa220005950 tx=0x7fa22000b410 comp rx=0 tx=0).stop 2023-07-20T19:06:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 msgr2=0x7fa234074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 0x7fa234074520 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa22800d980 tx=0x7fa22800dc90 comp rx=0 tx=0).stop 2023-07-20T19:06:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 shutdown_connections 2023-07-20T19:06:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7fa2240a1150 0x7fa2240a3610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2341022f0 0x7fa2340730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa234100760 0x7fa234074520 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.589 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 --2- 172.21.15.61:0/3165134559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa234101950 0x7fa234072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.589 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 >> 172.21.15.61:0/3165134559 conn(0x7fa2340fbcf0 msgr2=0x7fa234104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:47.589 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.527+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 shutdown_connections 2023-07-20T19:06:47.589 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.528+0000 7fa23ab4f700 1 -- 172.21.15.61:0/3165134559 wait complete. 2023-07-20T19:06:47.604 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T19:06:47.738 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 -- 172.21.15.61:0/3377729785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb00ff690 msgr2=0x7f3bb00ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:47.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/3377729785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb00ff690 0x7f3bb00ffab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40099c0 tx=0x7f3ba4009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 -- 172.21.15.61:0/3377729785 shutdown_connections 2023-07-20T19:06:47.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/3377729785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb0101220 0x7f3bb0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.739 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/3377729785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb00ff690 0x7f3bb00ffab0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/3377729785 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3bb0100880 0x7f3bb0100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.678+0000 7f3bb6af4700 1 -- 172.21.15.61:0/3377729785 >> 172.21.15.61:0/3377729785 conn(0x7f3bb00fac00 msgr2=0x7f3bb00fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 -- 172.21.15.61:0/3377729785 shutdown_connections 2023-07-20T19:06:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 -- 172.21.15.61:0/3377729785 wait complete. 2023-07-20T19:06:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 Processor -- start 2023-07-20T19:06:47.741 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 -- start start 2023-07-20T19:06:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb00ff690 0x7f3bb0103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3bb0100880 0x7f3bb0101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 0x7f3bb01021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0102940 con 0x7f3bb0101220 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0102ab0 con 0x7f3bb0100880 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb6af4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb01a0520 con 0x7f3bb00ff690 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.679+0000 7f3bb5091700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 0x7f3bb01021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 0x7f3bb01021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58536/0 (socket says 172.21.15.61:58536) 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 -- 172.21.15.61:0/318217035 learned_addr learned my addr 172.21.15.61:0/318217035 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb4890700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb00ff690 0x7f3bb0103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3baffff700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3bb0100880 0x7f3bb0101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 -- 172.21.15.61:0/318217035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb00ff690 msgr2=0x7f3bb0103620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb00ff690 0x7f3bb0103620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 -- 172.21.15.61:0/318217035 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3bb0100880 msgr2=0x7f3bb0101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3bb0100880 0x7f3bb0101c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 -- 172.21.15.61:0/318217035 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40095f0 con 0x7f3bb0101220 2023-07-20T19:06:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb5091700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 0x7f3bb01021a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800ada0 tx=0x7f3ba800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba800f070 con 0x7f3bb0101220 2023-07-20T19:06:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ba8007b30 con 0x7f3bb0101220 2023-07-20T19:06:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8013ba0 con 0x7f3bb0101220 2023-07-20T19:06:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb01a0800 con 0x7f3bb0101220 2023-07-20T19:06:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.680+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb01a0d10 con 0x7f3bb0101220 2023-07-20T19:06:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.681+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb010aef0 con 0x7f3bb0101220 2023-07-20T19:06:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.682+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3ba800c980 con 0x7f3bb0101220 2023-07-20T19:06:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.683+0000 7f3badffb700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3ba00a11d0 0x7f3ba00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.683+0000 7f3bb4890700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3ba00a11d0 0x7f3ba00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.683+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3ba80cc080 con 0x7f3bb0101220 2023-07-20T19:06:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.684+0000 7f3bb4890700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3ba00a11d0 0x7f3ba00a3690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400b580 tx=0x7f3ba4009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:47.748 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.687+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3ba80959b0 con 0x7f3bb0101220 2023-07-20T19:06:47.918 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-i[155514]: 2023-07-20T19:06:47.745+0000 7f275e353700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T19:06:47.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: pgmap v250: 97 pgs: 23 active+clean, 31 creating+peering, 43 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2023-07-20T19:06:47.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/253174943' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T19:06:47.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:47.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3165134559' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T19:06:47.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T19:06:47.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T19:06:47.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/3165134559' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T19:06:47.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T19:06:47.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:47 smithi156 ceph-mon[129228]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T19:06:47.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: pgmap v250: 97 pgs: 23 active+clean, 31 creating+peering, 43 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2023-07-20T19:06:47.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/253174943' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T19:06:47.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:47.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3165134559' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T19:06:47.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T19:06:47.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T19:06:47.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/3165134559' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T19:06:47.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T19:06:47.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:47 smithi061 ceph-mon[127454]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T19:06:47.994 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:47.935+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f3bb010b0b0 con 0x7f3bb0101220 2023-07-20T19:06:48.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: pgmap v250: 97 pgs: 23 active+clean, 31 creating+peering, 43 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2023-07-20T19:06:48.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/253174943' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T19:06:48.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-07-20T19:06:48.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3165134559' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T19:06:48.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T19:06:48.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T19:06:48.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/3165134559' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T19:06:48.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2023-07-20T19:06:48.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:47 smithi149 ceph-mon[128885]: daemon mds.b is now active in filesystem cephfs as rank 1 2023-07-20T19:06:48.590 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.531+0000 7f3badffb700 1 -- 172.21.15.61:0/318217035 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7f3ba8098fd0 con 0x7f3bb0101220 2023-07-20T19:06:48.593 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3ba00a11d0 msgr2=0x7f3ba00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:48.593 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3ba00a11d0 0x7f3ba00a3690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400b580 tx=0x7f3ba4009f10 comp rx=0 tx=0).stop 2023-07-20T19:06:48.594 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 msgr2=0x7f3bb01021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:48.594 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 0x7f3bb01021a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800ada0 tx=0x7f3ba800c330 comp rx=0 tx=0).stop 2023-07-20T19:06:48.594 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 shutdown_connections 2023-07-20T19:06:48.594 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f3ba00a11d0 0x7f3ba00a3690 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.596 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb0101220 0x7f3bb01021a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.597 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb00ff690 0x7f3bb0103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.597 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 --2- 172.21.15.61:0/318217035 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3bb0100880 0x7f3bb0101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.597 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 >> 172.21.15.61:0/318217035 conn(0x7f3bb00fac00 msgr2=0x7f3bb00688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:48.597 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 shutdown_connections 2023-07-20T19:06:48.597 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.534+0000 7f3bb6af4700 1 -- 172.21.15.61:0/318217035 wait complete. 2023-07-20T19:06:48.611 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T19:06:48.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 -- 172.21.15.61:0/3923275123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 msgr2=0x7f241c108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:48.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 --2- 172.21.15.61:0/3923275123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c108c00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f241400b0d0 tx=0x7f241400b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 -- 172.21.15.61:0/3923275123 shutdown_connections 2023-07-20T19:06:48.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 --2- 172.21.15.61:0/3923275123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c108c00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.745 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 --2- 172.21.15.61:0/3923275123 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f241c1007c0 0x7f241c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 --2- 172.21.15.61:0/3923275123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1019b0 0x7f241c101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 -- 172.21.15.61:0/3923275123 >> 172.21.15.61:0/3923275123 conn(0x7f241c0fbd50 msgr2=0x7f241c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 -- 172.21.15.61:0/3923275123 shutdown_connections 2023-07-20T19:06:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.685+0000 7f24212f6700 1 -- 172.21.15.61:0/3923275123 wait complete. 2023-07-20T19:06:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 Processor -- start 2023-07-20T19:06:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 -- start start 2023-07-20T19:06:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1007c0 0x7f241c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f241c1019b0 0x7f241c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c19e090 con 0x7f241c102350 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c19e1d0 con 0x7f241c1019b0 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f24212f6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c19e360 con 0x7f241c1007c0 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c19dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58550/0 (socket says 172.21.15.61:58550) 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 -- 172.21.15.61:0/2329883103 learned_addr learned my addr 172.21.15.61:0/2329883103 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241a7fc700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f241c1019b0 0x7f241c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241affd700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1007c0 0x7f241c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 -- 172.21.15.61:0/2329883103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1007c0 msgr2=0x7f241c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1007c0 0x7f241c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 -- 172.21.15.61:0/2329883103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f241c1019b0 msgr2=0x7f241c196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f241c1019b0 0x7f241c196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241b7fe700 1 -- 172.21.15.61:0/2329883103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414009d00 con 0x7f241c102350 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.686+0000 7f241affd700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1007c0 0x7f241c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T19:06:48.750 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.687+0000 7f241b7fe700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c19dac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2414008ec0 tx=0x7f2414008ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:48.751 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.687+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241400e040 con 0x7f241c102350 2023-07-20T19:06:48.751 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.687+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f241400be30 con 0x7f241c102350 2023-07-20T19:06:48.751 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.687+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241401b6c0 con 0x7f241c102350 2023-07-20T19:06:48.751 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.687+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c19e5e0 con 0x7f241c102350 2023-07-20T19:06:48.751 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.687+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c19e9b0 con 0x7f241c102350 2023-07-20T19:06:48.751 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.688+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c107d50 con 0x7f241c102350 2023-07-20T19:06:48.752 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.689+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2414005090 con 0x7f241c102350 2023-07-20T19:06:48.752 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.690+0000 7f2403fff700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f24040a1150 0x7f24040a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:48.752 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.690+0000 7f241affd700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f24040a1150 0x7f24040a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:48.752 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.690+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24140ce1f0 con 0x7f241c102350 2023-07-20T19:06:48.753 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.690+0000 7f241affd700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f24040a1150 0x7f24040a3610 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f241c101810 tx=0x7f240c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:48.753 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.693+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2414097a80 con 0x7f241c102350 2023-07-20T19:06:48.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/318217035' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T19:06:48.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.149:6838/284468426,v1:172.21.15.149:6839/284468426] up:active 2023-07-20T19:06:48.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/318217035' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T19:06:48.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T19:06:48.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2023-07-20T19:06:48.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2023-07-20T19:06:48.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:48 smithi156 ceph-mon[129228]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T19:06:48.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/318217035' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T19:06:48.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.149:6838/284468426,v1:172.21.15.149:6839/284468426] up:active 2023-07-20T19:06:48.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/318217035' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T19:06:48.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T19:06:48.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2023-07-20T19:06:48.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2023-07-20T19:06:48.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:48 smithi061 ceph-mon[127454]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T19:06:48.999 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:48.940+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f241c02c970 con 0x7f241c102350 2023-07-20T19:06:49.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/318217035' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T19:06:49.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.149:6838/284468426,v1:172.21.15.149:6839/284468426] up:active 2023-07-20T19:06:49.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/318217035' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T19:06:49.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T19:06:49.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2023-07-20T19:06:49.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2023-07-20T19:06:49.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:48 smithi149 ceph-mon[128885]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T19:06:49.596 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.537+0000 7f2403fff700 1 -- 172.21.15.61:0/2329883103 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2414011390 con 0x7f241c102350 2023-07-20T19:06:49.599 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f24040a1150 msgr2=0x7f24040a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f24040a1150 0x7f24040a3610 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f241c101810 tx=0x7f240c009260 comp rx=0 tx=0).stop 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 msgr2=0x7f241c19dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c19dac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2414008ec0 tx=0x7f2414008ef0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 shutdown_connections 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f24040a1150 0x7f24040a3610 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f241c102350 0x7f241c19dac0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.600 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1007c0 0x7f241c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.601 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 --2- 172.21.15.61:0/2329883103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f241c1019b0 0x7f241c196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.601 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 >> 172.21.15.61:0/2329883103 conn(0x7f241c0fbd50 msgr2=0x7f241c104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:49.601 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 shutdown_connections 2023-07-20T19:06:49.601 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.540+0000 7f24212f6700 1 -- 172.21.15.61:0/2329883103 wait complete. 2023-07-20T19:06:49.618 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-07-20T19:06:49.619 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T19:06:49.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 -- 172.21.15.61:0/1452398819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c100850 msgr2=0x7f5b9c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:49.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 --2- 172.21.15.61:0/1452398819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c100850 0x7f5b9c100cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b840099c0 tx=0x7f5b84009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 -- 172.21.15.61:0/1452398819 shutdown_connections 2023-07-20T19:06:49.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 --2- 172.21.15.61:0/1452398819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b9c1011f0 0x7f5b9c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 --2- 172.21.15.61:0/1452398819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.755 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 --2- 172.21.15.61:0/1452398819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c100850 0x7f5b9c100cb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.756 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 -- 172.21.15.61:0/1452398819 >> 172.21.15.61:0/1452398819 conn(0x7f5b9c0fabd0 msgr2=0x7f5b9c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:49.756 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 -- 172.21.15.61:0/1452398819 shutdown_connections 2023-07-20T19:06:49.756 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 -- 172.21.15.61:0/1452398819 wait complete. 2023-07-20T19:06:49.757 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.695+0000 7f5ba10de700 1 Processor -- start 2023-07-20T19:06:49.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 -- start start 2023-07-20T19:06:49.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:49.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b9c100850 0x7f5b9c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:49.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b9c1011f0 0x7f5b9c19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:49.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c19dff0 con 0x7f5b9c0ff660 2023-07-20T19:06:49.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c19e130 con 0x7f5b9c1011f0 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5ba10de700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c19e2c0 con 0x7f5b9c100850 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58560/0 (socket says 172.21.15.61:58560) 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 -- 172.21.15.61:0/2833314292 learned_addr learned my addr 172.21.15.61:0/2833314292 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9a59c700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b9c100850 0x7f5b9c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9b59e700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b9c1011f0 0x7f5b9c19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 -- 172.21.15.61:0/2833314292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b9c100850 msgr2=0x7f5b9c1969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:49.759 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b9c100850 0x7f5b9c1969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 -- 172.21.15.61:0/2833314292 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b9c1011f0 msgr2=0x7f5b9c19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b9c1011f0 0x7f5b9c19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 -- 172.21.15.61:0/2833314292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840095f0 con 0x7f5b9c0ff660 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.696+0000 7f5b9ad9d700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c196470 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00e970 tx=0x7f5b8c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.697+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c009790 con 0x7f5b9c0ff660 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.697+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b8c00c7e0 con 0x7f5b9c0ff660 2023-07-20T19:06:49.760 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.697+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9c19e5a0 con 0x7f5b9c0ff660 2023-07-20T19:06:49.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.697+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c003c70 con 0x7f5b9c0ff660 2023-07-20T19:06:49.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.697+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c19eab0 con 0x7f5b9c0ff660 2023-07-20T19:06:49.761 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.698+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9c10ae70 con 0x7f5b9c0ff660 2023-07-20T19:06:49.762 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.699+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5b8c00cc50 con 0x7f5b9c0ff660 2023-07-20T19:06:49.762 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.699+0000 7f5b93fff700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5b880a1150 0x7f5b880a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:49.762 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.699+0000 7f5b9a59c700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5b880a1150 0x7f5b880a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:49.762 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.699+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5b8c014070 con 0x7f5b9c0ff660 2023-07-20T19:06:49.762 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.700+0000 7f5b9a59c700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5b880a1150 0x7f5b880a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400b580 tx=0x7f5b84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:49.763 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.703+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5b8c096210 con 0x7f5b9c0ff660 2023-07-20T19:06:49.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: pgmap v251: 97 pgs: 68 active+clean, 23 creating+peering, 6 unknown; 580 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 10 op/s 2023-07-20T19:06:49.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2329883103' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T19:06:49.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6836/4080706577,v1:172.21.15.156:6837/4080706577] up:active 2023-07-20T19:06:49.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: from='client.? 172.21.15.61:0/2329883103' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T19:06:49.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T19:06:49.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2023-07-20T19:06:49.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2023-07-20T19:06:49.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:49 smithi156 ceph-mon[129228]: daemon mds.f is now active in filesystem cephfs as rank 3 2023-07-20T19:06:49.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: pgmap v251: 97 pgs: 68 active+clean, 23 creating+peering, 6 unknown; 580 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 10 op/s 2023-07-20T19:06:49.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2329883103' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T19:06:49.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6836/4080706577,v1:172.21.15.156:6837/4080706577] up:active 2023-07-20T19:06:49.970 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: from='client.? 172.21.15.61:0/2329883103' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T19:06:49.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T19:06:49.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2023-07-20T19:06:49.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2023-07-20T19:06:49.971 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-mon[127454]: daemon mds.f is now active in filesystem cephfs as rank 3 2023-07-20T19:06:49.992 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:49.933+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b9c068040 con 0x7f5b880a1150 2023-07-20T19:06:50.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: pgmap v251: 97 pgs: 68 active+clean, 23 creating+peering, 6 unknown; 580 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 10 op/s 2023-07-20T19:06:50.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2329883103' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T19:06:50.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6836/4080706577,v1:172.21.15.156:6837/4080706577] up:active 2023-07-20T19:06:50.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: from='client.? 172.21.15.61:0/2329883103' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T19:06:50.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T19:06:50.055 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2023-07-20T19:06:50.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2023-07-20T19:06:50.056 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:49 smithi149 ceph-mon[128885]: daemon mds.f is now active in filesystem cephfs as rank 3 2023-07-20T19:06:50.470 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:49.939+0000 7f7e960b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T19:06:50.470 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:49.939+0000 7f7e960b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T19:06:50.470 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:49.939+0000 7f7e960b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T19:06:50.471 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:49.939+0000 7f7e960b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T19:06:50.471 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:49.939+0000 7f7e960b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T19:06:50.623 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.563+0000 7f5b93fff700 1 -- 172.21.15.61:0/2833314292 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b9c068040 con 0x7f5b880a1150 2023-07-20T19:06:50.625 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5b880a1150 msgr2=0x7f5b880a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5b880a1150 0x7f5b880a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400b580 tx=0x7f5b84005fb0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 msgr2=0x7f5b9c196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c196470 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00e970 tx=0x7f5b8c00ec80 comp rx=0 tx=0).stop 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 shutdown_connections 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f5b880a1150 0x7f5b880a3610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b9c1011f0 0x7f5b9c19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.626 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b9c0ff660 0x7f5b9c196470 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.627 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 --2- 172.21.15.61:0/2833314292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b9c100850 0x7f5b9c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.627 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 >> 172.21.15.61:0/2833314292 conn(0x7f5b9c0fabd0 msgr2=0x7f5b9c067930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:50.627 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 shutdown_connections 2023-07-20T19:06:50.627 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.566+0000 7f5ba10de700 1 -- 172.21.15.61:0/2833314292 wait complete. 2023-07-20T19:06:50.640 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa '' 2023-07-20T19:06:50.779 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 -- 172.21.15.61:0/2573216553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52781011f0 msgr2=0x7f5278107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:50.779 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 --2- 172.21.15.61:0/2573216553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52781011f0 0x7f5278107aa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f527000b0d0 tx=0x7f527000b3e0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 -- 172.21.15.61:0/2573216553 shutdown_connections 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 --2- 172.21.15.61:0/2573216553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52781011f0 0x7f5278107aa0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 --2- 172.21.15.61:0/2573216553 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52780ff660 0x7f52780ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 --2- 172.21.15.61:0/2573216553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5278100850 0x7f5278100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 -- 172.21.15.61:0/2573216553 >> 172.21.15.61:0/2573216553 conn(0x7f52780fabd0 msgr2=0x7f52780fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 -- 172.21.15.61:0/2573216553 shutdown_connections 2023-07-20T19:06:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.719+0000 7f527ee31700 1 -- 172.21.15.61:0/2573216553 wait complete. 2023-07-20T19:06:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 Processor -- start 2023-07-20T19:06:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 -- start start 2023-07-20T19:06:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 0x7f5278068440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5278100850 0x7f527819b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52781011f0 0x7f527819d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527819e040 con 0x7f52780ff660 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527819e1b0 con 0x7f5278100850 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527ee31700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527819e370 con 0x7f52781011f0 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527cbcd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 0x7f5278068440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527cbcd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 0x7f5278068440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58592/0 (socket says 172.21.15.61:58592) 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527cbcd700 1 -- 172.21.15.61:0/206044941 learned_addr learned my addr 172.21.15.61:0/206044941 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f5277fff700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5278100850 0x7f527819b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.720+0000 7f527d3ce700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52781011f0 0x7f527819d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527cbcd700 1 -- 172.21.15.61:0/206044941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52781011f0 msgr2=0x7f527819d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527cbcd700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52781011f0 0x7f527819d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527cbcd700 1 -- 172.21.15.61:0/206044941 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5278100850 msgr2=0x7f527819b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527cbcd700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5278100850 0x7f527819b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527cbcd700 1 -- 172.21.15.61:0/206044941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270009d00 con 0x7f52780ff660 2023-07-20T19:06:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527cbcd700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 0x7f5278068440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c970 tx=0x7f526c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c004220 con 0x7f52780ff660 2023-07-20T19:06:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527819e650 con 0x7f52780ff660 2023-07-20T19:06:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f526c004380 con 0x7f52780ff660 2023-07-20T19:06:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c003b60 con 0x7f52780ff660 2023-07-20T19:06:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.721+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527819eb60 con 0x7f52780ff660 2023-07-20T19:06:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.722+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5278106590 con 0x7f52780ff660 2023-07-20T19:06:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.723+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f526c003ce0 con 0x7f52780ff660 2023-07-20T19:06:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.724+0000 7f5275ffb700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f52680a1150 0x7f52680a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.724+0000 7f5277fff700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f52680a1150 0x7f52680a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.724+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f526c0cbf10 con 0x7f52780ff660 2023-07-20T19:06:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.724+0000 7f5277fff700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f52680a1150 0x7f52680a3610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5264005950 tx=0x7f526400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:50.788 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.728+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f526c0957b0 con 0x7f52780ff660 2023-07-20T19:06:51.024 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.964+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f5278060bb0 con 0x7f52680a1150 2023-07-20T19:06:51.048 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.989+0000 7f5275ffb700 1 -- 172.21.15.61:0/206044941 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5278060bb0 con 0x7f52680a1150 2023-07-20T19:06:51.050 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f52680a1150 msgr2=0x7f52680a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f52680a1150 0x7f52680a3610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5264005950 tx=0x7f526400a300 comp rx=0 tx=0).stop 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 msgr2=0x7f5278068440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 0x7f5278068440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c970 tx=0x7f526c00cc80 comp rx=0 tx=0).stop 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 shutdown_connections 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f52680a1150 0x7f52680a3610 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52781011f0 0x7f527819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52780ff660 0x7f5278068440 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.052 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 --2- 172.21.15.61:0/206044941 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5278100850 0x7f527819b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.052 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 >> 172.21.15.61:0/206044941 conn(0x7f52780fabd0 msgr2=0x7f52780fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:51.052 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 shutdown_connections 2023-07-20T19:06:51.052 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:50.991+0000 7f527ee31700 1 -- 172.21.15.61:0/206044941 wait complete. 2023-07-20T19:06:51.063 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2023-07-20T19:06:51.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 -- 172.21.15.61:0/1974293063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc0ff660 msgr2=0x7f29cc0ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:51.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 --2- 172.21.15.61:0/1974293063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc0ff660 0x7f29cc0ffa80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f29c00099c0 tx=0x7f29c0009cd0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 -- 172.21.15.61:0/1974293063 shutdown_connections 2023-07-20T19:06:51.193 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 --2- 172.21.15.61:0/1974293063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc1011f0 0x7f29cc107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 --2- 172.21.15.61:0/1974293063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc0ff660 0x7f29cc0ffa80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 --2- 172.21.15.61:0/1974293063 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f29cc100850 0x7f29cc100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 -- 172.21.15.61:0/1974293063 >> 172.21.15.61:0/1974293063 conn(0x7f29cc0fabd0 msgr2=0x7f29cc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T19:06:51.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 -- 172.21.15.61:0/1974293063 shutdown_connections 2023-07-20T19:06:51.194 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.133+0000 7f29d11ec700 1 -- 172.21.15.61:0/1974293063 wait complete. 2023-07-20T19:06:51.196 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.134+0000 7f29d11ec700 1 Processor -- start 2023-07-20T19:06:51.196 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.134+0000 7f29d11ec700 1 -- start start 2023-07-20T19:06:51.196 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.134+0000 7f29d11ec700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc0ff660 0x7f29cc1953b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:51.196 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.134+0000 7f29d11ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc100850 0x7f29cc1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.134+0000 7f29d11ec700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f29cc1011f0 0x7f29cc19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29d11ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19cf30 con 0x7f29cc100850 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29d11ec700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19d070 con 0x7f29cc1011f0 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29d11ec700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19d200 con 0x7f29cc0ff660 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc100850 0x7f29cc1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc100850 0x7f29cc1958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58610/0 (socket says 172.21.15.61:58610) 2023-07-20T19:06:51.197 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 -- 172.21.15.61:0/1521960468 learned_addr learned my addr 172.21.15.61:0/1521960468 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29cad9d700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc0ff660 0x7f29cc1953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29cb59e700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f29cc1011f0 0x7f29cc19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 -- 172.21.15.61:0/1521960468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc0ff660 msgr2=0x7f29cc1953b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc0ff660 0x7f29cc1953b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 -- 172.21.15.61:0/1521960468 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f29cc1011f0 msgr2=0x7f29cc19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f29cc1011f0 0x7f29cc19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 -- 172.21.15.61:0/1521960468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c00095f0 con 0x7f29cc100850 2023-07-20T19:06:51.198 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29cb59e700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f29cc1011f0 0x7f29cc19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29ca59c700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29cc100850 0x7f29cc1958f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00d950 tx=0x7f29bc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc009940 con 0x7f29cc100850 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29d11ec700 1 -- 172.21.15.61:0/1521960468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc19d4e0 con 0x7f29cc100850 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29bc010460 con 0x7f29cc100850 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc00f620 con 0x7f29cc100850 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.135+0000 7f29d11ec700 1 -- 172.21.15.61:0/1521960468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc19da40 con 0x7f29cc100850 2023-07-20T19:06:51.199 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.136+0000 7f29d11ec700 1 -- 172.21.15.61:0/1521960468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc106610 con 0x7f29cc100850 2023-07-20T19:06:51.200 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.137+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f29bc0108d0 con 0x7f29cc100850 2023-07-20T19:06:51.200 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.138+0000 7f29b3fff700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f29b40a1150 0x7f29b40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T19:06:51.200 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.138+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f29bc0cc650 con 0x7f29cc100850 2023-07-20T19:06:51.200 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.138+0000 7f29cad9d700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f29b40a1150 0x7f29b40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T19:06:51.201 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.138+0000 7f29cad9d700 1 --2- 172.21.15.61:0/1521960468 >> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] conn(0x7f29b40a1150 0x7f29b40a3610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f29c000b580 tx=0x7f29c0009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T19:06:51.201 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.142+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f29bc095ef0 con 0x7f29cc100850 2023-07-20T19:06:51.437 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.377+0000 7f29d11ec700 1 -- 172.21.15.61:0/1521960468 --> [v2:172.21.15.61:6800/3980156535,v1:172.21.15.61:6801/3980156535] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f29cc060c40 con 0x7f29b40a1150 2023-07-20T19:06:51.439 INFO:teuthology.orchestra.run.smithi061.stderr:2023-07-20T19:06:51.380+0000 7f29b3fff700 1 -- 172.21.15.61:0/1521960468 <== mgr.14152 v2:172.21.15.61:6800/3980156535 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7f29cc060c40 con 0x7f29b40a1150 2023-07-20T19:06:51.441 INFO:teuthology.orchestra.run.smithi061.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T19:06:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T19:06:51.445 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi061 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T19:06:51.446 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:06:51.473 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:06:51.509 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:06:51.547 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T19:06:51.548 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T19:06:51.565 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T19:06:51.584 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T19:06:51.607 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T19:06:51.608 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T19:06:51.608 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a 2023-07-20T19:06:51.720 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:51.380+0000 7f7e960b8700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T19:06:51.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: pgmap v252: 97 pgs: 97 active+clean; 580 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 12 op/s 2023-07-20T19:06:51.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:06:51.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: mds.? [v2:172.21.15.156:6834/1691160499,v1:172.21.15.156:6835/1691160499] up:active 2023-07-20T19:06:51.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T19:06:51.721 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T19:06:51.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: Cluster is now healthy 2023-07-20T19:06:51.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2023-07-20T19:06:51.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T19:06:51.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-mon[127454]: daemon mds.h is now active in filesystem cephfs as rank 4 2023-07-20T19:06:51.722 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 systemd[1]: Stopping Ceph mon.a for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:51.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: pgmap v252: 97 pgs: 97 active+clean; 580 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 12 op/s 2023-07-20T19:06:51.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:06:51.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: mds.? [v2:172.21.15.156:6834/1691160499,v1:172.21.15.156:6835/1691160499] up:active 2023-07-20T19:06:51.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T19:06:51.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T19:06:51.805 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: Cluster is now healthy 2023-07-20T19:06:51.806 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2023-07-20T19:06:51.806 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T19:06:51.806 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:51 smithi149 ceph-mon[128885]: daemon mds.h is now active in filesystem cephfs as rank 4 2023-07-20T19:06:51.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: pgmap v252: 97 pgs: 97 active+clean; 580 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 12 op/s 2023-07-20T19:06:51.918 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T19:06:51.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: mds.? [v2:172.21.15.156:6834/1691160499,v1:172.21.15.156:6835/1691160499] up:active 2023-07-20T19:06:51.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T19:06:51.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T19:06:51.919 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: Cluster is now healthy 2023-07-20T19:06:51.920 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2023-07-20T19:06:51.920 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-20T19:06:51.920 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:51 smithi156 ceph-mon[129228]: daemon mds.h is now active in filesystem cephfs as rank 4 2023-07-20T19:06:52.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-a[127430]: 2023-07-20T19:06:51.894+0000 7f4be0c75700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:06:52.220 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-a[127430]: 2023-07-20T19:06:51.894+0000 7f4be0c75700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T19:06:52.635 INFO:journalctl@ceph.mon.a.smithi061.stdout:Jul 20 19:06:52 smithi061 bash[163761]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-a 2023-07-20T19:06:52.643 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T19:06:52.740 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:06:52.741 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T19:06:52.741 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T19:06:52.741 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.b 2023-07-20T19:06:53.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:52 smithi149 systemd[1]: Stopping Ceph mon.b for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:53.167 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:06:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-i[155514]: 2023-07-20T19:06:52.744+0000 7f275e353700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-20T19:06:53.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-b[128861]: 2023-07-20T19:06:53.159+0000 7f6336b37700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:06:53.555 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-b[128861]: 2023-07-20T19:06:53.159+0000 7f6336b37700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T19:06:53.954 INFO:journalctl@ceph.mon.b.smithi149.stdout:Jul 20 19:06:53 smithi149 bash[155185]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-b 2023-07-20T19:06:53.962 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T19:06:54.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:06:54.030 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T19:06:54.030 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T19:06:54.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.c 2023-07-20T19:06:54.402 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:54 smithi156 systemd[1]: Stopping Ceph mon.c for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:54.660 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-c[129201]: 2023-07-20T19:06:54.342+0000 7f8cc72ed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:06:54.660 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-c[129201]: 2023-07-20T19:06:54.342+0000 7f8cc72ed700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 20 19:06:54 smithi156 bash[156907]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mon-c 2023-07-20T19:06:55.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T19:06:55.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:06:55.074 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T19:06:55.075 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T19:06:55.076 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T19:06:55.430 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:55 smithi061 systemd[1]: Stopping Ceph mgr.x for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:55.719 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:55 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x[127704]: 2023-07-20T19:06:55.370+0000 7f7f27cac700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:06:55.969 INFO:journalctl@ceph.mgr.x.smithi061.stdout:Jul 20 19:06:55 smithi061 bash[164277]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T19:06:55.975 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T19:06:56.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:06:56.017 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T19:06:56.017 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T19:06:56.017 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T19:06:56.304 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:06:55 smithi149 systemd[1]: Stopping Ceph mgr.y for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:56.904 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:06:56 smithi149 bash[155531]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T19:06:56.904 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:06:56 smithi149 systemd[1]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T19:06:56.909 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T19:06:56.934 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:06:56 smithi149 systemd[1]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.y.service: Failed with result 'exit-code'. 2023-07-20T19:06:56.934 INFO:journalctl@ceph.mgr.y.smithi149.stdout:Jul 20 19:06:56 smithi149 systemd[1]: Stopped Ceph mgr.y for 7541eb30-272f-11ee-9b34-001a4aab830c. 2023-07-20T19:06:56.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:06:56.948 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T19:06:56.949 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T19:06:56.949 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T19:06:57.383 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:06:56 smithi156 systemd[1]: Stopping Ceph mgr.z for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:57.659 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:06:57 smithi156 bash[157105]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T19:06:57.843 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 20 19:06:57 smithi156 systemd[1]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T19:06:57.847 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T19:06:57.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:06:57.884 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T19:06:57.885 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T19:06:57.885 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.0 2023-07-20T19:06:58.220 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:06:57 smithi061 systemd[1]: Stopping Ceph osd.0 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:06:58.720 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:06:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0[140435]: 2023-07-20T19:06:58.186+0000 7fba35956700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:06:58.721 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:06:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0[140435]: 2023-07-20T19:06:58.186+0000 7fba35956700 -1 osd.0 81 *** Got signal Terminated *** 2023-07-20T19:06:58.721 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:06:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0[140435]: 2023-07-20T19:06:58.186+0000 7fba35956700 -1 osd.0 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:04.030 INFO:journalctl@ceph.osd.0.smithi061.stdout:Jul 20 19:07:03 smithi061 bash[164415]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-0 2023-07-20T19:07:05.609 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T19:07:05.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:05.652 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T19:07:05.653 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T19:07:05.653 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.1 2023-07-20T19:07:06.019 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:07:05 smithi149 systemd[1]: Stopping Ceph osd.1 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:06.305 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:07:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1[133856]: 2023-07-20T19:07:05.958+0000 7f21cfdde700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:06.305 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:07:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1[133856]: 2023-07-20T19:07:05.958+0000 7f21cfdde700 -1 osd.1 81 *** Got signal Terminated *** 2023-07-20T19:07:06.305 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:07:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1[133856]: 2023-07-20T19:07:05.958+0000 7f21cfdde700 -1 osd.1 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:12.055 INFO:journalctl@ceph.osd.1.smithi149.stdout:Jul 20 19:07:11 smithi149 bash[155676]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-1 2023-07-20T19:07:13.168 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T19:07:13.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:13.208 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T19:07:13.209 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T19:07:13.209 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.2 2023-07-20T19:07:13.565 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:07:13 smithi156 systemd[1]: Stopping Ceph osd.2 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:13.918 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:07:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2[135068]: 2023-07-20T19:07:13.505+0000 7efe7d3ce700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:13.918 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:07:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2[135068]: 2023-07-20T19:07:13.505+0000 7efe7d3ce700 -1 osd.2 81 *** Got signal Terminated *** 2023-07-20T19:07:13.919 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:07:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2[135068]: 2023-07-20T19:07:13.505+0000 7efe7d3ce700 -1 osd.2 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:19.418 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 20 19:07:19 smithi156 bash[157252]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-2 2023-07-20T19:07:20.588 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T19:07:20.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:20.630 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T19:07:20.630 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T19:07:20.631 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.3 2023-07-20T19:07:20.971 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:07:20 smithi061 systemd[1]: Stopping Ceph osd.3 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:21.471 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:07:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:07:20.932+0000 7fb1a3db6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:21.471 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:07:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:07:20.932+0000 7fb1a3db6700 -1 osd.3 81 *** Got signal Terminated *** 2023-07-20T19:07:21.471 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:07:20 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:07:20.932+0000 7fb1a3db6700 -1 osd.3 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:25.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:24.663+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:25.168 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:24.905+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:25.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:24 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:24.794+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:25.646 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:25 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:25.253+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:25.971 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:07:25 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3[145763]: 2023-07-20T19:07:25.585+0000 7fb19e3a9700 -1 osd.3 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.293935+0000 front 2023-07-20T19:07:00.293968+0000 (oldest deadline 2023-07-20T19:07:24.994079+0000) 2023-07-20T19:07:25.971 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:25 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:25.788+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:26.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:25.652+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:26.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:25.874+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:26.568 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:26.241+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:26.874 INFO:journalctl@ceph.osd.3.smithi061.stdout:Jul 20 19:07:26 smithi061 bash[164682]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-3 2023-07-20T19:07:27.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:26 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:26.658+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:27.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:26 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:26.879+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:27.225 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:26 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:26.813+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:27.712 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:27 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:27.197+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:27.949 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:27 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:27.628+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:27.971 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:27 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:27.813+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:28.114 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T19:07:28.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:28.156 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T19:07:28.157 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T19:07:28.157 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.4 2023-07-20T19:07:28.418 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:27 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:27.889+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:28.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:28 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:28.222+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:28.543 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:07:28 smithi149 systemd[1]: Stopping Ceph osd.4 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:28.543 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:28 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:28.034+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:28.805 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:07:28 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4[139303]: 2023-07-20T19:07:28.482+0000 7fadb24ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:28.805 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:07:28 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4[139303]: 2023-07-20T19:07:28.482+0000 7fadb24ec700 -1 osd.4 81 *** Got signal Terminated *** 2023-07-20T19:07:28.806 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:07:28 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4[139303]: 2023-07-20T19:07:28.482+0000 7fadb24ec700 -1 osd.4 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:29.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:28 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:28.652+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:29.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:28 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:28.872+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:29.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:28 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:28.858+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:29.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:28 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:28.846+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:29.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:29 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:29.004+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:29.720 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:29 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:29.229+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:29.957 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:29 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:29.631+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:29.957 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:29 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:29.861+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:30.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:29 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:29.801+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:30.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:29 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:29.986+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:30.418 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:29 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:29.893+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:30.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:30 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:30.240+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:30.998 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:30 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:30.678+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:30.998 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:30 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:30.815+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:31.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:30 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:30.785+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:31.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:30 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:30.963+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:31.418 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:30 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:30.938+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:31.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:31 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:31.216+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:32.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:31 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:31.858+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:32.170 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:31 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:31.703+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:32.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:31 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:31.940+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:32.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:31 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:31.834+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:32.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:32 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:32.011+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:32.720 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:32 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:32.254+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:33.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:32 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:32.859+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:33.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:32 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:32.732+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:33.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:32 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:32.909+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:33.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:32 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:32.909+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:33.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:32 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:32.818+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:33.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:33 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:33.004+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:33.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:33 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:33.217+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:34.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:33 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:33.721+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:34.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:33 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:33.901+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:34.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:33 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:33.901+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:34.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:33 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:33.926+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:34.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:33 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:33.926+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:34.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:33 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:33.828+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:34.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:33 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:33.828+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:34.305 INFO:journalctl@ceph.osd.4.smithi149.stdout:Jul 20 19:07:34 smithi149 bash[155952]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-4 2023-07-20T19:07:34.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:34 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:34.057+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:34.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:33 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:33.962+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:34.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:33 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:33.962+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:34.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:34 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:34.245+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:34.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:34 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:34.245+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:35.055 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:34 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:34.938+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:35.056 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:34 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:34.938+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:35.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:34 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:34.854+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:35.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:34 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:34.854+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:35.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:34 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:34.757+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:35.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:34 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:34.757+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:35.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:34 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:34.897+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:35.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:34 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:34.897+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:35.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:34 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:34.804+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:35.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:34 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:34.804+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:35.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:35 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:35.020+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:35.661 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T19:07:35.702 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:35.703 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T19:07:35.704 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T19:07:35.705 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.5 2023-07-20T19:07:35.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:35 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:35.229+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:35.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:35 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:35.229+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:36.061 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:35 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:35.810+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:36.061 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:35 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:35.810+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:36.061 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:35 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:35.878+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:36.062 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:35 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:35.878+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:36.062 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:35 smithi156 systemd[1]: Stopping Ceph osd.5 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:36.062 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:35 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:35.743+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:36.062 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:35 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:35.743+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:36.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:35 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:35.811+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:36.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:35 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:35.812+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:36.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:36 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:36.055+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:36.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:35 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:35.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:36.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:35 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:35.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:36.418 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:36.001+0000 7f24e6037700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:36.418 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:36.001+0000 7f24e6037700 -1 osd.5 81 *** Got signal Terminated *** 2023-07-20T19:07:36.419 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:36.001+0000 7f24e6037700 -1 osd.5 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:36.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:36 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:36.194+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:36.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:36 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:36.194+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:37.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:36.704+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:37.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:36.704+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:37.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:36.800+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:37.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:36.800+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:37.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:36.856+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:37.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:36 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:36.856+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:37.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:36 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:36.844+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:37.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:36 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:36.844+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:37.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:37 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:37.057+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:37.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:36 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:36.936+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:37.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:36 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:36.936+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:37.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:37 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:37.242+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:37.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:37 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:37.242+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:38.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:37 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:37.692+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:38.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:37 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:37.692+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:38.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:37 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:37.826+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:38.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:37 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:37.826+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:38.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:37 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:37.830+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:38.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:37 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:37.830+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:38.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:37 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:37.797+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:38.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:37 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:37.797+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:38.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:38 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:38.066+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:38.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:37 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:37.945+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:38.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:37 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:37.945+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:38.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:38 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:38.215+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:38.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:38 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:38.215+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:39.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:38 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:38.686+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:39.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:38 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:38.686+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:39.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:38 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:38.779+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:39.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:38 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:38.779+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:39.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:38 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:38.815+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:39.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:38 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:38.815+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:39.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:38 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:38.791+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:39.221 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:38 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:38.791+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:39.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:39 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:39.025+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:39.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:38 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:38.982+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:39.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:38 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:38.982+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:39.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:39.184+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:39.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:39.184+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:40.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:39.714+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:40.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:39.714+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:40.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:39.714+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.652530+0000 front 2023-07-20T19:07:16.652502+0000 (oldest deadline 2023-07-20T19:07:38.952665+0000) 2023-07-20T19:07:40.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:39.845+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:40.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:39.845+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:40.171 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:39.845+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:40.171 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:39.740+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:40.171 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:39.740+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:40.172 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:39 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:39.740+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:40.216 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:39.829+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:40.217 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:39 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:39.829+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:40.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:40 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:40.070+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:40.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:40 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:40.070+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:40.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:39 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:39.981+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:40.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:39 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:39.981+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:40.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:40.156+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:40.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:40.156+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:40.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:40.156+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:41.168 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:40.690+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.549747+0000 front 2023-07-20T19:06:58.549680+0000 (oldest deadline 2023-07-20T19:07:24.449846+0000) 2023-07-20T19:07:41.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:40.690+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.951118+0000 front 2023-07-20T19:07:07.951016+0000 (oldest deadline 2023-07-20T19:07:33.851082+0000) 2023-07-20T19:07:41.169 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5[140574]: 2023-07-20T19:07:40.690+0000 7f24de626700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.652530+0000 front 2023-07-20T19:07:16.652502+0000 (oldest deadline 2023-07-20T19:07:38.952665+0000) 2023-07-20T19:07:41.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:40.788+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:41.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:40.788+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:41.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:40.788+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:41.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:40.796+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:41.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:40.796+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:41.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:40 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:40.796+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:41.199 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:40.803+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:41.200 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:40 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:40.803+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:41.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:41 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:41.054+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:41.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:41 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:41.054+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:41.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:40 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:40.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:41.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:40 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:40.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:41.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:41.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:41.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:41.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:41.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:41.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:41.919 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 20 19:07:41 smithi156 bash[157519]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-5 2023-07-20T19:07:41.919 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:41 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:41.783+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:41.919 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:41 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:41.783+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:41.920 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:41 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:41.783+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:41.920 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:41 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:41.776+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:41.920 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:41 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:41.776+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:41.920 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:41 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:41.776+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:42.188 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:41.759+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:42.188 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:41 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:41.759+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:42.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:42.127+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:42.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:42.127+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:42.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:42.127+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:42.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:42 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:42.035+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:42.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:42 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:42.035+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:42.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:42 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:42.010+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:42.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:42 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:42.010+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:43.105 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:42 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:42.832+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:43.105 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:42 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:42.832+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:43.105 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:42 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:42.832+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:43.105 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:42 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:42.794+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:43.106 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:42 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:42.794+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:43.106 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:42 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:42.794+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:43.112 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T19:07:43.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:43.154 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T19:07:43.154 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T19:07:43.155 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.6 2023-07-20T19:07:43.167 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:42.734+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:43.168 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:42.734+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:43.471 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:43 smithi061 systemd[1]: Stopping Ceph osd.6 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:43.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:43.107+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:43.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:43.107+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:43.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:43.107+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:43.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:43 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:43.022+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:43.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:43 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:43.022+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:43.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:43 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:43.023+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:43.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:43 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:43.023+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:43.826 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:43.454+0000 7f6928645700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:43.826 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:43.454+0000 7f6928645700 -1 osd.6 81 *** Got signal Terminated *** 2023-07-20T19:07:43.827 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:43.454+0000 7f6928645700 -1 osd.6 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:43.827 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:43.766+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:44.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:43 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:43.790+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:44.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:43 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:43.790+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:44.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:43 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:43.790+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:44.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:43 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:43.773+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:44.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:43 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:43.773+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:44.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:43 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:43.773+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:44.210 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:43 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:43.766+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:44.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:44.149+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:44.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:44.149+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:44.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:44.149+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:44.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:44 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:44.047+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:44.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:44 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:44.047+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:44.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:44 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:44.025+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:44.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:44 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:44.025+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:44.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:44 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:44.025+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:45.164 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:44.717+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:45.165 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:44.717+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:45.165 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:44 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:44.717+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.746189+0000 front 2023-07-20T19:07:15.746384+0000 (oldest deadline 2023-07-20T19:07:44.546824+0000) 2023-07-20T19:07:45.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:44 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:44.771+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:45.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:44 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:44.771+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:45.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:44 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:44.771+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:45.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:44 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:44.761+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:45.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:44 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:44.761+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:45.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:44 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:44.761+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:45.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:45.105+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:45.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:45.105+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:45.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:45.105+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:45.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:45 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:45.082+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:45.556 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:45 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:45.082+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:45.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:45 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:45.037+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:45.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:45 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:45.037+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:45.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:45 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:45.037+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:46.158 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:45.758+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:46.159 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:45.758+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:46.159 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:45 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:45.758+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.746189+0000 front 2023-07-20T19:07:15.746384+0000 (oldest deadline 2023-07-20T19:07:44.546824+0000) 2023-07-20T19:07:46.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:45 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:45.729+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:46.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:45 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:45.729+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:46.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:45 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:45.729+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:46.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:45 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:45.806+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:46.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:45 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:45.806+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:46.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:45 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:45.806+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:46.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:46.097+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:46.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:46.097+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:46.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:46.097+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:46.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:46.071+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:46.556 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:46.071+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:46.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:46.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:46.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:46.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:46.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:46.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:47.147 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:46.727+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:47.148 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:46.727+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:47.148 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:46 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:46.727+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.746189+0000 front 2023-07-20T19:07:15.746384+0000 (oldest deadline 2023-07-20T19:07:44.546824+0000) 2023-07-20T19:07:47.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:46 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:46.705+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:47.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:46 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:46.705+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:47.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:46 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:46.705+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:47.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:46 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:46.839+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:47.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:46 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:46.839+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:47.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:46 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:46.839+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:47.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:47.027+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:47.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:47.027+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:47.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:47.027+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:47.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:46.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:47.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:46.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:47.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:46 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:46.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:47.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:47.087+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:47.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:47.087+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:47.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:47.087+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:48.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:47.692+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:48.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:47.692+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:48.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:47.692+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:48.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:47.873+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:48.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:47.873+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:48.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:47 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:47.873+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:48.184 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:47.775+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:03.043119+0000 front 2023-07-20T19:07:03.043153+0000 (oldest deadline 2023-07-20T19:07:24.143241+0000) 2023-07-20T19:07:48.185 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:47.775+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:10.545128+0000 front 2023-07-20T19:07:10.545071+0000 (oldest deadline 2023-07-20T19:07:32.845062+0000) 2023-07-20T19:07:48.185 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:47 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6[151186]: 2023-07-20T19:07:47.775+0000 7f6921435700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.746189+0000 front 2023-07-20T19:07:15.746384+0000 (oldest deadline 2023-07-20T19:07:44.546824+0000) 2023-07-20T19:07:48.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:48 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:48.042+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:48.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:48 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:48.042+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:48.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:48 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:48.042+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:48.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:47.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:48.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:47.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:48.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:47.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:48.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:48 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:48.124+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:48.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:48 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:48.124+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:48.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:48 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:48.124+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:49.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:48.659+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:49.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:48.659+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:49.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:48.659+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:49.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:48.903+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:49.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:48.903+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:49.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:48 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:48.903+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:49.471 INFO:journalctl@ceph.osd.6.smithi061.stdout:Jul 20 19:07:49 smithi061 bash[164959]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-6 2023-07-20T19:07:49.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:49.110+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:49.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:49.110+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:49.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:49 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:49.110+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:49.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:49.053+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:49.556 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:49.053+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:49.556 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:49.053+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:49.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.002+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:49.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.002+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:49.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.002+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:50.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:49.658+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:50.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:49.658+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:50.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:49.658+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:50.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:49.658+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:50.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:49.889+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:50.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:49.889+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:50.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:49 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:49.889+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:50.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:50.074+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:50.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:50.074+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:50.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:50.074+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:50.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.981+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:50.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.981+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:50.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.981+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:50.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:49 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:49.981+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:50.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:50 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:50.129+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:50.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:50 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:50.129+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:50.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:50 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:50.129+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:50.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:50 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:50.129+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:50.609 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T19:07:50.691 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:50.692 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T19:07:50.692 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T19:07:50.692 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.7 2023-07-20T19:07:51.041 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:50 smithi149 systemd[1]: Stopping Ceph osd.7 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:51.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:50.873+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:51.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:50.873+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:51.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:50.873+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:51.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:50.679+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:51.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:50.679+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:51.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:50.679+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:51.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:50 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:50.679+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:51.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:50.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:51.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:50.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:51.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:50.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:51.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:50 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:50.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:51.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:51.000+0000 7f6855ede700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:51.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:51.000+0000 7f6855ede700 -1 osd.7 81 *** Got signal Terminated *** 2023-07-20T19:07:51.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:51.000+0000 7f6855ede700 -1 osd.7 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:51.307 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:51.068+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:51.307 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:51.068+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:51.307 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:51.068+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:51.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:51.150+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:51.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:51.150+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:51.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:51.150+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:51.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:51 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:51.150+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:52.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:51.722+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:52.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:51.722+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:52.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:51.722+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:52.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:51.722+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:52.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:51.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:52.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:51.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:52.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:51 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:51.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:52.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:52.043+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:52.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:52.043+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:52.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:52.043+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:52.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:51.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:52.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:51.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:52.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:51.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:52.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:51 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:51.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:52.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:52 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:52.122+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:52.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:52 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:52.122+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:52.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:52 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:52.122+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:52.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:52 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:52.122+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:52.992 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:52.677+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:52.993 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:52.677+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:52.993 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:52.677+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:52.993 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:52.677+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:53.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:53.008+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:53.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:53.008+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:53.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:53.008+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:53.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:52.937+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:53.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:52.937+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:53.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:52.937+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:53.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:52 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:52.937+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:53.418 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:52.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:53.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:52.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:53.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:52 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:52.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:53.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:53 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:53.140+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:53.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:53 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:53.140+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:53.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:53 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:53.140+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:53.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:53 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:53.140+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:54.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:53.693+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:54.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:53.693+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:54.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:53.693+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:54.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:53.693+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:54.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:53.893+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:54.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:53.893+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:54.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:53 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:53.893+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:54.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:54.027+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:54.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:54.027+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:54.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:54.027+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:54.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:53.982+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:54.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:53.982+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:54.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:53.982+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:54.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:53.982+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:54.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:54 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:54.183+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:54.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:54 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:54.183+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:54.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:54 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:54.183+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:54.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:54 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:54.183+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:55.168 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:54.670+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:55.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:54.670+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:55.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:54.670+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:55.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:54.670+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:55.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:54.870+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:55.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:54.870+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:55.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:54 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:54.870+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:55.305 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:55.023+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.839466+0000 front 2023-07-20T19:07:09.839494+0000 (oldest deadline 2023-07-20T19:07:33.339613+0000) 2023-07-20T19:07:55.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:55.023+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:17.941256+0000 front 2023-07-20T19:07:17.941097+0000 (oldest deadline 2023-07-20T19:07:39.041325+0000) 2023-07-20T19:07:55.306 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7[144721]: 2023-07-20T19:07:55.023+0000 7f684e4cd700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:22.542544+0000 front 2023-07-20T19:07:22.542622+0000 (oldest deadline 2023-07-20T19:07:46.642449+0000) 2023-07-20T19:07:55.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:54.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:55.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:54.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:55.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:54.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:55.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:54 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:54.980+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:55.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:55 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:55.159+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:55.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:55 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:55.159+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:55.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:55 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:55.159+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:55.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:55 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:55.159+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:56.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:55.630+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:56.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:55.630+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:56.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:55.630+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:56.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:55.630+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:56.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:55.824+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:56.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:55.824+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:56.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:55.824+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:56.170 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:55 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:55.824+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:07:56.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:55.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:56.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:55.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:56.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:55.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:56.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:55 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:55.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:56.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:56 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:56.186+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:56.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:56 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:56.186+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:56.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:56 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:56.186+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:56.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:56 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:56.186+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:56.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:56 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:56.186+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:07:56.929 INFO:journalctl@ceph.osd.7.smithi149.stdout:Jul 20 19:07:56 smithi149 bash[156213]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-7 2023-07-20T19:07:57.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:56.867+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:57.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:56.867+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:57.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:56.867+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:57.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:56.867+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:07:57.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:56.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:57.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:56.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:57.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:56.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:57.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:56.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:57.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:56 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:56.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:07:57.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:57 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:57.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:57.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:57 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:57.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:57.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:57 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:57.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:57.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:57 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:57.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:57.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:57 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:57.139+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:07:57.498 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:57.498 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:57.498 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:57.498 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:57.498 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.005+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:07:57.968 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:57.625+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:57.968 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:57.625+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:57.968 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:57.625+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:57.968 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:57.625+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:57.968 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:57.625+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:07:58.289 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:58.290 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:58.290 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:58.290 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:58.290 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:57 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:57.961+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:07:58.296 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T19:07:58.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:07:58.338 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T19:07:58.338 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T19:07:58.339 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.8 2023-07-20T19:07:58.368 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:57.908+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:58.369 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:57.908+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:58.369 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:57.908+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:58.369 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:57 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:57.908+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:07:58.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:58.161+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:58.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:58.161+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:58.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:58.161+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:58.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:58.161+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:58.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:58 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:58.161+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:07:58.669 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 systemd[1]: Stopping Ceph osd.8 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:07:59.015 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.627+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:07:59.016 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.627+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:07:59.016 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.627+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:07:59.016 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.627+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:07:59.016 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.627+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:07:59.016 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.664+0000 7f4fb2bc7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:07:59.016 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.664+0000 7f4fb2bc7700 -1 osd.8 81 *** Got signal Terminated *** 2023-07-20T19:07:59.017 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:58.664+0000 7f4fb2bc7700 -1 osd.8 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:07:59.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:58 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:58.970+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:07:59.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:58 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:58.970+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:07:59.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:58 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:58.970+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:07:59.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:58 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:58.970+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:07:59.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:07:58 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:07:58.970+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:07:59.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:58.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:07:59.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:58.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:07:59.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:58.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:07:59.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:58 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:58.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:07:59.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:59.175+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:07:59.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:59.175+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:07:59.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:59.175+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:07:59.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:59.175+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:07:59.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:07:59 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:07:59.175+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:00.029 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:59.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:08:00.029 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:59.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:08:00.029 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:59.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:08:00.029 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:59.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:08:00.029 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:07:59.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:08:00.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:59.968+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:00.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:59.968+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:00.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:59.968+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:00.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:07:59 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:07:59.968+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:00.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:00.014+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:00.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:00.014+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:00.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:00.014+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:00.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:00.014+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:00.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:00.014+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:00.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:00.177+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:00.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:00.177+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:00.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:00.177+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:00.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:00.177+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:00.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:00 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:00.177+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:01.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:00.612+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:08:01.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:00.612+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:08:01.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:00.612+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:08:01.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:00.612+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:08:01.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:00.612+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:08:01.418 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:00.939+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:01.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:00.939+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:01.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:00.939+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:01.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:00 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:00.939+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:01.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:01 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:01.049+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:01.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:01 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:01.049+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:01.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:01 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:01.049+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:01.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:01 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:01.049+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:01.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:01 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:01.049+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:01.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:01.222+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:01.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:01.222+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:01.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:01.222+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:01.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:01.222+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:01.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:01 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:01.222+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:01.975 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:01.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:08:01.975 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:01.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:08:01.975 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:01.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:08:01.976 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:01.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:08:01.976 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:01.624+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:08:02.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:01.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:02.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:01.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:02.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:01.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:02.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:01 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:01.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:02.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:02 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:02.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:02.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:02 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:02.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:02.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:02 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:02.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:02.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:02 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:02.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:02.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:02 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:02.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:02.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:02.220+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:02.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:02.220+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:02.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:02.220+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:02.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:02.221+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:02.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:02 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:02.221+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:03.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:02.872+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:03.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:02.872+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:03.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:02.872+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:03.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:02.872+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:03.169 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:02.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:58.555168+0000 front 2023-07-20T19:06:58.555194+0000 (oldest deadline 2023-07-20T19:07:27.955840+0000) 2023-07-20T19:08:03.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:02.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.057203+0000 front 2023-07-20T19:07:09.057015+0000 (oldest deadline 2023-07-20T19:07:33.757179+0000) 2023-07-20T19:08:03.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:02.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:13.757511+0000 front 2023-07-20T19:07:13.757486+0000 (oldest deadline 2023-07-20T19:07:39.657652+0000) 2023-07-20T19:08:03.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:02.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:25.860258+0000 front 2023-07-20T19:07:25.860478+0000 (oldest deadline 2023-07-20T19:07:48.760301+0000) 2023-07-20T19:08:03.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:02.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:31.661637+0000 front 2023-07-20T19:07:31.661679+0000 (oldest deadline 2023-07-20T19:07:55.762758+0000) 2023-07-20T19:08:03.170 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:02 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8[146092]: 2023-07-20T19:08:02.626+0000 7f4fad9bb700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:36.263670+0000 front 2023-07-20T19:07:36.263692+0000 (oldest deadline 2023-07-20T19:08:02.163776+0000) 2023-07-20T19:08:03.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:03 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:03.065+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:03.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:03 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:03.065+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:03.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:03 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:03.065+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:03.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:03 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:03.065+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:03.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:03 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:03.065+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:03.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:03 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:03.188+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:03.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:03 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:03.188+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:03.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:03 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:03.188+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:03.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:03 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:03.188+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:03.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:03 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:03.188+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:04.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:03 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:03.856+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:04.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:03 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:03.856+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:04.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:03 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:03.856+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:04.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:03 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:03.856+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:04.478 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 20 19:08:04 smithi156 bash[157788]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-8 2023-07-20T19:08:04.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:04 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:04.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:04.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:04 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:04.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:04.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:04 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:04.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:04.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:04 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:04.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:04.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:04 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:04.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:04.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:04 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:04.016+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:04.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:04.236+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:04.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:04.236+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:04.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:04.236+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:04.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:04.236+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:04.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:04 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:04.236+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:05.394 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:04 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:04.880+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:05.394 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:04 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:04.880+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:05.394 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:04 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:04.880+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:05.395 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:04 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:04.880+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:05.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:05.051+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:05.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:05.051+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:05.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:05.051+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:05.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:05.051+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:05.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:05.051+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:05.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:05 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:05.051+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:05.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:05.260+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:05.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:05.260+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:05.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:05.260+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:05.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:05.260+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:05.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:05.260+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:05.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:05.260+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:40.935080+0000 front 2023-07-20T19:07:40.935153+0000 (oldest deadline 2023-07-20T19:08:05.035049+0000) 2023-07-20T19:08:05.746 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T19:08:05.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:05.827 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T19:08:05.828 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T19:08:05.828 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.9 2023-07-20T19:08:06.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:05 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:05.921+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:06.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:05 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:05.921+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:06.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:05 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:05.921+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:06.169 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:05 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:05.921+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:06.187 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:05 smithi061 systemd[1]: Stopping Ceph osd.9 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:06.471 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.127+0000 7f34c116f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.127+0000 7f34c116f700 -1 osd.9 81 *** Got signal Terminated *** 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.127+0000 7f34c116f700 -1 osd.9 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.246+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.246+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.246+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.246+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:06.472 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.246+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:06.473 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:06 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:06.246+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:40.935080+0000 front 2023-07-20T19:07:40.935153+0000 (oldest deadline 2023-07-20T19:08:05.035049+0000) 2023-07-20T19:08:06.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:06 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:06.035+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:06.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:06 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:06.035+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:06.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:06 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:06.035+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:06.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:06 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:06.035+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:06.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:06 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:06.035+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:06.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:06 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:06.035+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:07.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:06 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:06.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:07.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:06 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:06.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:07.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:06 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:06.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:07.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:06 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:06.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:07.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:06 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:06.914+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:07.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:07 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:07.074+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:07.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:07 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:07.074+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:07.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:07 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:07.074+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:07.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:07 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:07.074+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:07.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:07 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:07.074+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:07.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:07 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:07.074+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:07.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:07 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:07.294+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:07.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:07 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:07.294+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:07.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:07 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:07.294+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:07.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:07 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:07.294+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:07.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:07 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:07.294+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:07.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:07 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:07.294+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:40.935080+0000 front 2023-07-20T19:07:40.935153+0000 (oldest deadline 2023-07-20T19:08:05.035049+0000) 2023-07-20T19:08:08.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:07 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:07.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:08.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:07 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:07.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:08.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:07 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:07.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:08.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:07 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:07.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:08.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:07 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:07.955+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:08.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:08 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:08.041+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:08.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:08 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:08.041+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:08.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:08 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:08.041+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:08.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:08 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:08.041+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:08.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:08 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:08.041+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:08.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:08 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:08.041+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:08.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:08.252+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:08.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:08.252+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:08.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:08.252+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:08.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:08.252+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:08.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:08.252+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:08.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:08 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:08.252+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:40.935080+0000 front 2023-07-20T19:07:40.935153+0000 (oldest deadline 2023-07-20T19:08:05.035049+0000) 2023-07-20T19:08:09.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:08 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:08.935+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:09.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:08 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:08.935+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:09.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:08 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:08.935+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:09.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:08 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:08.935+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:09.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:08 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:08.935+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:09.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:09 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:09.080+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:09.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:09 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:09.080+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:09.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:09 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:09.080+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:09.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:09 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:09.080+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:09.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:09 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:09.080+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:09.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:09 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:09.080+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:09.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:09.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:09.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:09.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:09.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:09.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:40.935080+0000 front 2023-07-20T19:07:40.935153+0000 (oldest deadline 2023-07-20T19:08:05.035049+0000) 2023-07-20T19:08:09.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:09 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:09.206+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:45.035503+0000 front 2023-07-20T19:07:45.035448+0000 (oldest deadline 2023-07-20T19:08:09.135535+0000) 2023-07-20T19:08:10.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:09 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:09.936+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:10.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:09 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:09.936+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:10.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:09 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:09.936+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:10.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:09 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:09.936+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:10.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:09 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:09.936+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:10.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:10 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:10.078+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:10.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:10 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:10.078+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:10.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:10 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:10.078+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:10.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:10 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:10.078+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:10.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:10 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:10.078+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:10.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:10 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:10.078+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:10.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:00.426283+0000 front 2023-07-20T19:07:00.426348+0000 (oldest deadline 2023-07-20T19:07:25.126510+0000) 2023-07-20T19:08:10.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:08.428975+0000 front 2023-07-20T19:07:08.428943+0000 (oldest deadline 2023-07-20T19:07:33.728977+0000) 2023-07-20T19:08:10.721 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:16.530944+0000 front 2023-07-20T19:07:16.530884+0000 (oldest deadline 2023-07-20T19:07:40.030901+0000) 2023-07-20T19:08:10.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:24.731997+0000 front 2023-07-20T19:07:24.731965+0000 (oldest deadline 2023-07-20T19:07:50.032162+0000) 2023-07-20T19:08:10.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.032791+0000 front 2023-07-20T19:07:30.032815+0000 (oldest deadline 2023-07-20T19:07:55.232974+0000) 2023-07-20T19:08:10.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:40.935080+0000 front 2023-07-20T19:07:40.935153+0000 (oldest deadline 2023-07-20T19:08:05.035049+0000) 2023-07-20T19:08:10.722 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:10 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9[156643]: 2023-07-20T19:08:10.170+0000 7f34b975e700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:45.035503+0000 front 2023-07-20T19:07:45.035448+0000 (oldest deadline 2023-07-20T19:08:09.135535+0000) 2023-07-20T19:08:11.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:10.946+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:11.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:10.946+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:11.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:10.946+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:11.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:10.946+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:11.421 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:10.946+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:11.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:11.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:11.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:11.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:11.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:11.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:11.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:11 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:11.042+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:12.221 INFO:journalctl@ceph.osd.9.smithi061.stdout:Jul 20 19:08:11 smithi061 bash[165228]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-9 2023-07-20T19:08:12.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:11 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:11.900+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:12.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:11 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:11.900+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:12.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:11 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:11.900+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:12.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:11 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:11.900+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:12.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:11 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:11.900+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:12.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:11 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:11.900+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:12.555 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:12.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:12.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:12.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:12.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:12.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:12.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:12 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:12.029+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:13.258 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T19:08:13.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:13.299 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T19:08:13.300 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T19:08:13.300 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.10 2023-07-20T19:08:13.330 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:13.330 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:13.330 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:13.331 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:13.331 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:13.331 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:13.331 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.008+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:13.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:12 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:12.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:13.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:12 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:12.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:13.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:12 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:12.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:13.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:12 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:12.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:13.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:12 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:12.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:13.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:12 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:12.933+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:13.657 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 systemd[1]: Stopping Ceph osd.10 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:13.658 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.597+0000 7f1ab8801700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:14.045 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.597+0000 7f1ab8801700 -1 osd.10 81 *** Got signal Terminated *** 2023-07-20T19:08:14.046 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.597+0000 7f1ab8801700 -1 osd.10 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:08:14.305 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:14.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:14.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:14.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:14.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:14.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:14.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:13 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:13.985+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:14.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:13.932+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:14.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:13.932+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:14.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:13.932+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:14.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:13.932+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:14.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:13.932+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:14.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:13 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:13.932+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:15.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:15.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:15.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:15.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:15.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:15.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:15.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:14 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:14.969+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:15.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:14.937+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:15.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:14.937+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:15.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:14.937+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:15.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:14.937+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:15.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:14.937+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:15.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:14 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:14.937+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:16.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:16.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:16.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:16.306 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:16.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:16.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:16.307 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:15 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:15.989+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:16.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:15.964+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:16.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:15.964+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:16.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:15.964+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:16.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:15.964+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:16.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:15.964+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:16.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:15.964+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:17.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:16 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:16.990+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:17.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:16 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:16.990+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:17.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:16 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:16.990+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:17.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:16 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:16.990+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:17.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:16 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:16.990+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:17.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:16 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:16.990+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:17.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:17.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:17.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:17.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:17.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:17.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:17.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:17 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:17.020+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:18.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:18.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:18.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:18.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:18.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:18.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:18.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:17 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:17.986+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:18.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:07:02.465318+0000 front 2023-07-20T19:07:02.465196+0000 (oldest deadline 2023-07-20T19:07:27.765276+0000) 2023-07-20T19:08:18.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:07.765689+0000 front 2023-07-20T19:07:07.765729+0000 (oldest deadline 2023-07-20T19:07:33.065803+0000) 2023-07-20T19:08:18.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:18.167847+0000 front 2023-07-20T19:07:18.167952+0000 (oldest deadline 2023-07-20T19:07:43.468084+0000) 2023-07-20T19:08:18.556 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6814 osd.3 since back 2023-07-20T19:07:23.468630+0000 front 2023-07-20T19:07:23.468611+0000 (oldest deadline 2023-07-20T19:07:49.368578+0000) 2023-07-20T19:08:18.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:32.269509+0000 front 2023-07-20T19:07:32.269559+0000 (oldest deadline 2023-07-20T19:07:56.969675+0000) 2023-07-20T19:08:18.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:39.270688+0000 front 2023-07-20T19:07:39.270755+0000 (oldest deadline 2023-07-20T19:08:03.370584+0000) 2023-07-20T19:08:18.557 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:18 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10[150165]: 2023-07-20T19:08:18.026+0000 7f1ab4df8700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:47.372317+0000 front 2023-07-20T19:07:47.372207+0000 (oldest deadline 2023-07-20T19:08:10.772725+0000) 2023-07-20T19:08:19.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:19.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:19.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:19.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:19.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:19.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:19.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.035+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:19.436 INFO:journalctl@ceph.osd.10.smithi149.stdout:Jul 20 19:08:19 smithi149 bash[156476]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-10 2023-07-20T19:08:20.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:20.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:20.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:20.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:20.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:20.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:20.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:19 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:19.988+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:20.701 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T19:08:20.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:20.781 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T19:08:20.782 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T19:08:20.782 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.11 2023-07-20T19:08:21.164 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 systemd[1]: Stopping Ceph osd.11 for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:21.164 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:21.164 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:21.164 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:21.164 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:21.165 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:21.165 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:21.165 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:20 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:20.961+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:21.165 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.103+0000 7f302b97c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:21.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.103+0000 7f302b97c700 -1 osd.11 81 *** Got signal Terminated *** 2023-07-20T19:08:21.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.103+0000 7f302b97c700 -1 osd.11 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T19:08:22.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:22.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:22.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:22.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:22.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:22.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:22.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:21.942+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:23.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:23.421 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:23.421 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:23.421 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:23.421 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:23.422 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:23.422 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:22 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:22.904+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:24.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:24.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:24.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:24.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:24.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:24.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:24.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:23 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:23.895+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:25.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:25.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:25.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:25.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:25.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:25.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:25.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:25.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:24 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:24.861+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2023-07-20T19:08:00.530333+0000 front 2023-07-20T19:08:00.530352+0000 (oldest deadline 2023-07-20T19:08:24.630339+0000) 2023-07-20T19:08:26.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6806 osd.0 since back 2023-07-20T19:06:59.920429+0000 front 2023-07-20T19:06:59.920399+0000 (oldest deadline 2023-07-20T19:07:24.620427+0000) 2023-07-20T19:08:26.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6804 osd.1 since back 2023-07-20T19:07:09.321931+0000 front 2023-07-20T19:07:09.321679+0000 (oldest deadline 2023-07-20T19:07:32.821793+0000) 2023-07-20T19:08:26.419 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-20T19:07:15.722876+0000 front 2023-07-20T19:07:15.722905+0000 (oldest deadline 2023-07-20T19:07:39.223073+0000) 2023-07-20T19:08:26.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6812 osd.4 since back 2023-07-20T19:07:30.126486+0000 front 2023-07-20T19:07:30.126463+0000 (oldest deadline 2023-07-20T19:07:55.426555+0000) 2023-07-20T19:08:26.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-20T19:07:35.426914+0000 front 2023-07-20T19:07:35.427069+0000 (oldest deadline 2023-07-20T19:08:06.027730+0000) 2023-07-20T19:08:26.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.61:6822 osd.6 since back 2023-07-20T19:07:46.028267+0000 front 2023-07-20T19:07:46.028393+0000 (oldest deadline 2023-07-20T19:08:11.328172+0000) 2023-07-20T19:08:26.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.149:6820 osd.7 since back 2023-07-20T19:07:53.029049+0000 front 2023-07-20T19:07:53.029097+0000 (oldest deadline 2023-07-20T19:08:17.729087+0000) 2023-07-20T19:08:26.420 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:25 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11[151680]: 2023-07-20T19:08:25.901+0000 7f3023f6b700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2023-07-20T19:08:00.530333+0000 front 2023-07-20T19:08:00.530352+0000 (oldest deadline 2023-07-20T19:08:24.630339+0000) 2023-07-20T19:08:26.823 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 20 19:08:26 smithi156 bash[158049]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-osd-11 2023-07-20T19:08:28.288 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T19:08:28.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:28.328 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T19:08:28.329 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T19:08:28.329 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.a 2023-07-20T19:08:28.693 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:08:28 smithi061 systemd[1]: Stopping Ceph mds.a for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:28.971 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:08:28 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-a[161923]: 2023-07-20T19:08:28.631+0000 7f2c7b9a6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:28.972 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:08:28 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-a[161923]: 2023-07-20T19:08:28.631+0000 7f2c7b9a6700 -1 mds.a *** got signal Terminated *** 2023-07-20T19:08:32.934 INFO:journalctl@ceph.mds.a.smithi061.stdout:Jul 20 19:08:32 smithi061 bash[165505]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-a 2023-07-20T19:08:32.941 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T19:08:32.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:32.979 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T19:08:32.980 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T19:08:32.980 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.d 2023-07-20T19:08:33.221 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:08:32 smithi061 systemd[1]: Stopping Ceph mds.d for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:33.722 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:08:33 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-d[160916]: 2023-07-20T19:08:33.306+0000 7efcc795e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:33.723 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:08:33 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-d[160916]: 2023-07-20T19:08:33.306+0000 7efcc795e700 -1 mds.d *** got signal Terminated *** 2023-07-20T19:08:37.471 INFO:journalctl@ceph.mds.d.smithi061.stdout:Jul 20 19:08:36 smithi061 bash[165652]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-d 2023-07-20T19:08:37.478 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T19:08:37.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:37.515 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T19:08:37.516 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T19:08:37.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.g 2023-07-20T19:08:37.908 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:08:37 smithi061 systemd[1]: Stopping Ceph mds.g for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:38.222 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:08:37 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-g[161536]: 2023-07-20T19:08:37.846+0000 7f08934fb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:38.222 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:08:37 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-g[161536]: 2023-07-20T19:08:37.846+0000 7f08934fb700 -1 mds.g *** got signal Terminated *** 2023-07-20T19:08:42.589 INFO:journalctl@ceph.mds.g.smithi061.stdout:Jul 20 19:08:42 smithi061 bash[165805]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-g 2023-07-20T19:08:42.594 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T19:08:42.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:42.630 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T19:08:42.630 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T19:08:42.631 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.j 2023-07-20T19:08:42.972 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:08:42 smithi061 systemd[1]: Stopping Ceph mds.j for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:43.472 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:08:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-j[160550]: 2023-07-20T19:08:42.955+0000 7f1dcd082700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:43.472 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:08:42 smithi061 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-j[160550]: 2023-07-20T19:08:42.955+0000 7f1dcd082700 -1 mds.j *** got signal Terminated *** 2023-07-20T19:08:47.148 INFO:journalctl@ceph.mds.j.smithi061.stdout:Jul 20 19:08:46 smithi061 bash[165952]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-j 2023-07-20T19:08:47.235 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T19:08:47.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:47.272 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T19:08:47.272 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T19:08:47.273 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.b 2023-07-20T19:08:47.556 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:08:47 smithi149 systemd[1]: Stopping Ceph mds.b for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:48.057 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:08:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-b[153955]: 2023-07-20T19:08:47.575+0000 7fab89ec9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:48.057 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:08:47 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-b[153955]: 2023-07-20T19:08:47.575+0000 7fab89ec9700 -1 mds.b *** got signal Terminated *** 2023-07-20T19:08:53.221 INFO:journalctl@ceph.mds.b.smithi149.stdout:Jul 20 19:08:52 smithi149 bash[156741]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-b 2023-07-20T19:08:53.228 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T19:08:53.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:53.266 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T19:08:53.267 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T19:08:53.267 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.e 2023-07-20T19:08:53.557 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:08:53 smithi149 systemd[1]: Stopping Ceph mds.e for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:54.056 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:08:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-e[153266]: 2023-07-20T19:08:53.590+0000 7f4c82697700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:54.056 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:08:53 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-e[153266]: 2023-07-20T19:08:53.590+0000 7f4c82697700 -1 mds.e *** got signal Terminated *** 2023-07-20T19:08:55.734 INFO:journalctl@ceph.mds.e.smithi149.stdout:Jul 20 19:08:55 smithi149 bash[156890]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-e 2023-07-20T19:08:55.740 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T19:08:55.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:08:55.775 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T19:08:55.776 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T19:08:55.776 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.h 2023-07-20T19:08:56.056 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:08:55 smithi149 systemd[1]: Stopping Ceph mds.h for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:08:56.556 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:08:56 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-h[153615]: 2023-07-20T19:08:56.099+0000 7f90800f6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:08:56.557 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:08:56 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-h[153615]: 2023-07-20T19:08:56.099+0000 7f90800f6700 -1 mds.h *** got signal Terminated *** 2023-07-20T19:09:00.126 INFO:journalctl@ceph.mds.h.smithi149.stdout:Jul 20 19:09:00 smithi149 bash[157033]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-h 2023-07-20T19:09:00.311 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T19:09:00.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:09:00.345 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T19:09:00.346 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T19:09:00.346 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.k 2023-07-20T19:09:00.734 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:09:00 smithi149 systemd[1]: Stopping Ceph mds.k for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:09:01.056 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:09:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-k[152924]: 2023-07-20T19:09:00.673+0000 7f075f939700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:09:01.057 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:09:00 smithi149 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-k[152924]: 2023-07-20T19:09:00.673+0000 7f075f939700 -1 mds.k *** got signal Terminated *** 2023-07-20T19:09:05.198 INFO:journalctl@ceph.mds.k.smithi149.stdout:Jul 20 19:09:05 smithi149 bash[157175]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-k 2023-07-20T19:09:05.396 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T19:09:05.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:09:05.432 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T19:09:05.432 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T19:09:05.433 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.c 2023-07-20T19:09:05.808 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:09:05 smithi156 systemd[1]: Stopping Ceph mds.c for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:09:06.169 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:09:05 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-c[155174]: 2023-07-20T19:09:05.746+0000 7f42e911d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:09:06.170 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:09:05 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-c[155174]: 2023-07-20T19:09:05.746+0000 7f42e911d700 -1 mds.c *** got signal Terminated *** 2023-07-20T19:09:10.312 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 20 19:09:09 smithi156 bash[158311]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-c 2023-07-20T19:09:10.318 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T19:09:10.357 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:09:10.358 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T19:09:10.358 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T19:09:10.359 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.f 2023-07-20T19:09:10.670 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:09:10 smithi156 systemd[1]: Stopping Ceph mds.f for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:09:11.170 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:09:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-f[154828]: 2023-07-20T19:09:10.669+0000 7fc78fbe1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:09:11.170 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:09:10 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-f[154828]: 2023-07-20T19:09:10.669+0000 7fc78fbe1700 -1 mds.f *** got signal Terminated *** 2023-07-20T19:09:15.090 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 20 19:09:14 smithi156 bash[158458]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-f 2023-07-20T19:09:15.096 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T19:09:15.130 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:09:15.131 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T19:09:15.132 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T19:09:15.132 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.i 2023-07-20T19:09:15.420 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:09:15 smithi156 systemd[1]: Stopping Ceph mds.i for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:09:15.920 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:09:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-i[155514]: 2023-07-20T19:09:15.443+0000 7f276235b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:09:15.920 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:09:15 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-i[155514]: 2023-07-20T19:09:15.443+0000 7f276235b700 -1 mds.i *** got signal Terminated *** 2023-07-20T19:09:21.170 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 20 19:09:20 smithi156 bash[158598]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-i 2023-07-20T19:09:21.186 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T19:09:21.220 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:09:21.221 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T19:09:21.221 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T19:09:21.222 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.l 2023-07-20T19:09:21.622 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:09:21 smithi156 systemd[1]: Stopping Ceph mds.l for 7541eb30-272f-11ee-9b34-001a4aab830c... 2023-07-20T19:09:21.920 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:09:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-l[154486]: 2023-07-20T19:09:21.560+0000 7fc28c707700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T19:09:21.920 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:09:21 smithi156 ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-l[154486]: 2023-07-20T19:09:21.560+0000 7fc28c707700 -1 mds.l *** got signal Terminated *** 2023-07-20T19:09:26.170 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 20 19:09:25 smithi156 bash[158738]: ceph-7541eb30-272f-11ee-9b34-001a4aab830c-mds-l 2023-07-20T19:09:26.214 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7541eb30-272f-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T19:09:26.248 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T19:09:26.249 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T19:09:26.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T19:09:26.574 INFO:teuthology.orchestra.run.smithi061.stdout:Deleting cluster with fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:09:28.215 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T19:09:28.524 INFO:teuthology.orchestra.run.smithi149.stdout:Deleting cluster with fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:09:30.100 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T19:09:30.399 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:09:31.972 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:09:32.011 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:09:32.047 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T19:09:32.076 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T19:09:32.079 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi061/crash 2023-07-20T19:09:32.081 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T19:09:32.128 INFO:teuthology.orchestra.run.smithi061.stderr:tar: /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T19:09:32.128 INFO:teuthology.orchestra.run.smithi061.stderr:tar: Error is not recoverable: exiting now 2023-07-20T19:09:32.135 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi149/crash 2023-07-20T19:09:32.136 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T19:09:32.179 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T19:09:32.179 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now 2023-07-20T19:09:32.181 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi156/crash 2023-07-20T19:09:32.184 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T19:09:32.219 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T19:09:32.219 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2023-07-20T19:09:32.220 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T19:09:32.221 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-07-20T19:09:32.255 INFO:teuthology.orchestra.run.smithi061.stderr:grep: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T19:09:32.257 INFO:tasks.cephadm:Compressing logs... 2023-07-20T19:09:32.258 DEBUG:teuthology.orchestra.run.smithi061:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T19:09:32.301 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T19:09:32.304 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T19:09:32.329 INFO:teuthology.orchestra.run.smithi061.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T19:09:32.330 INFO:teuthology.orchestra.run.smithi061.stderr:: No such file or directory 2023-07-20T19:09:32.330 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T19:09:32.330 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T19:09:32.330 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T19:09:32.331 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.g.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T19:09:32.331 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T19:09:32.332 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T19:09:32.334 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T19:09:32.334 INFO:teuthology.orchestra.run.smithi156.stderr:: No such file or directory 2023-07-20T19:09:32.335 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T19:09:32.335 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T19:09:32.335 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.5.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T19:09:32.336 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.8.log 2023-07-20T19:09:32.336 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T19:09:32.340 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.8.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.f.log: 91.3% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T19:09:32.340 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T19:09:32.341 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T19:09:32.342 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.d.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.x.log: 94.9% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T19:09:32.342 INFO:teuthology.orchestra.run.smithi061.stderr: 94.0% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T19:09:32.342 INFO:teuthology.orchestra.run.smithi061.stderr: 93.8% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T19:09:32.342 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T19:09:32.343 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T19:09:32.343 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T19:09:32.343 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T19:09:32.343 INFO:teuthology.orchestra.run.smithi061.stderr: 94.8% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T19:09:32.343 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T19:09:32.344 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.c.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.i.log: 93.4% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T19:09:32.344 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T19:09:32.344 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.10.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T19:09:32.345 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T19:09:32.346 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.k.log: /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log: 94.1% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T19:09:32.346 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T19:09:32.346 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T19:09:32.346 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.c.log: 93.5% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T19:09:32.347 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/cephadm.log: 91.6% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T19:09:32.347 INFO:teuthology.orchestra.run.smithi061.stderr: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T19:09:32.347 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T19:09:32.349 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.b.log: 94.1% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T19:09:32.350 INFO:teuthology.orchestra.run.smithi149.stderr: 93.5% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T19:09:32.350 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T19:09:32.350 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.b.log: 91.5% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T19:09:32.350 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T19:09:32.351 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T19:09:32.352 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T19:09:32.352 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/cephadm.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T19:09:32.353 INFO:teuthology.orchestra.run.smithi149.stderr: 93.3% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T19:09:32.354 INFO:teuthology.orchestra.run.smithi149.stderr: 91.1% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T19:09:32.358 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.l.log: 91.5% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T19:09:32.358 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T19:09:32.358 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T19:09:32.359 INFO:teuthology.orchestra.run.smithi156.stderr: 94.0% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T19:09:32.368 INFO:teuthology.orchestra.run.smithi156.stderr: 93.4% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T19:09:32.423 INFO:teuthology.orchestra.run.smithi156.stderr: 92.8% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T19:09:32.424 INFO:teuthology.orchestra.run.smithi061.stderr: 88.7% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T19:09:32.454 INFO:teuthology.orchestra.run.smithi156.stderr: 92.1% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T19:09:32.459 INFO:teuthology.orchestra.run.smithi061.stderr: 92.0% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T19:09:32.466 INFO:teuthology.orchestra.run.smithi149.stderr: 91.9% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T19:09:32.470 INFO:teuthology.orchestra.run.smithi149.stderr: 92.2% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T19:09:32.483 INFO:teuthology.orchestra.run.smithi061.stderr: 92.2% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T19:09:32.495 INFO:teuthology.orchestra.run.smithi149.stderr: 92.0% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T19:09:32.512 INFO:teuthology.orchestra.run.smithi061.stderr: 91.8% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T19:09:32.520 INFO:teuthology.orchestra.run.smithi156.stderr: 92.2% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T19:09:32.525 INFO:teuthology.orchestra.run.smithi156.stderr: 91.8% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T19:09:32.533 INFO:teuthology.orchestra.run.smithi061.stderr: 92.0% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T19:09:32.535 INFO:teuthology.orchestra.run.smithi156.stderr: 91.8% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T19:09:32.535 INFO:teuthology.orchestra.run.smithi149.stderr: 92.0% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T19:09:32.536 INFO:teuthology.orchestra.run.smithi156.stderr: 2023-07-20T19:09:32.537 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m0.216s 2023-07-20T19:09:32.537 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m0.833s 2023-07-20T19:09:32.537 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.046s 2023-07-20T19:09:32.547 INFO:teuthology.orchestra.run.smithi149.stderr: 91.7% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T19:09:32.549 INFO:teuthology.orchestra.run.smithi149.stderr: 2023-07-20T19:09:32.549 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m0.228s 2023-07-20T19:09:32.550 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m0.828s 2023-07-20T19:09:32.550 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.067s 2023-07-20T19:09:32.689 INFO:teuthology.orchestra.run.smithi061.stderr: 91.1% -- replaced with /var/log/ceph/7541eb30-272f-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T19:09:32.689 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-07-20T19:09:32.690 INFO:teuthology.orchestra.run.smithi061.stderr:real 0m0.372s 2023-07-20T19:09:32.690 INFO:teuthology.orchestra.run.smithi061.stderr:user 0m1.099s 2023-07-20T19:09:32.690 INFO:teuthology.orchestra.run.smithi061.stderr:sys 0m0.062s 2023-07-20T19:09:32.690 INFO:tasks.cephadm:Archiving logs... 2023-07-20T19:09:32.691 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi061/log 2023-07-20T19:09:32.696 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T19:09:32.915 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi149/log 2023-07-20T19:09:32.918 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T19:09:33.081 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi156/log 2023-07-20T19:09:33.082 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T19:09:33.230 INFO:tasks.cephadm:Removing cluster... 2023-07-20T19:09:33.231 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --force 2023-07-20T19:09:33.526 INFO:teuthology.orchestra.run.smithi061.stdout:Deleting cluster with fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:09:33.990 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --force 2023-07-20T19:09:34.296 INFO:teuthology.orchestra.run.smithi149.stdout:Deleting cluster with fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:09:34.751 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7541eb30-272f-11ee-9b34-001a4aab830c --force 2023-07-20T19:09:35.053 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 7541eb30-272f-11ee-9b34-001a4aab830c 2023-07-20T19:09:35.503 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T19:09:35.504 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T19:09:35.526 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T19:09:35.545 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T19:09:35.564 INFO:tasks.cephadm:Teardown complete 2023-07-20T19:09:35.565 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi061 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T19:09:35.655 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5aea0705bbf84ad9b39b50b522198299 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_78296bf9274b016ce68ff036026e51a56af22f7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi061 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T19:09:35.662 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T19:09:35.672 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T19:09:35.687 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T19:09:35.689 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T19:09:40.742 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T19:09:40.746 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T19:09:50.894 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T19:09:50.912 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T19:09:50.913 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T19:09:50.920 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T19:09:50.925 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T19:09:50.934 INFO:teuthology.orchestra.run.smithi061.stderr:bash: ntpq: command not found 2023-07-20T19:09:50.939 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T19:09:50.940 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-07-20T19:09:50.940 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +1058us[+1054us] +/- 86ms 2023-07-20T19:09:50.940 INFO:teuthology.orchestra.run.smithi061.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -2041us[-2045us] +/- 52ms 2023-07-20T19:09:50.940 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1703us[+1699us] +/- 66ms 2023-07-20T19:09:50.941 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T19:09:50.941 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2023-07-20T19:09:50.946 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T19:09:50.946 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2023-07-20T19:09:50.947 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +1069us[+1068us] +/- 86ms 2023-07-20T19:09:50.947 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -2044us[-2045us] +/- 52ms 2023-07-20T19:09:50.948 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1691us[+1690us] +/- 66ms 2023-07-20T19:09:50.948 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T19:09:50.949 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-07-20T19:09:50.951 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T19:09:50.952 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-07-20T19:09:50.952 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +1076us[+1076us] +/- 86ms 2023-07-20T19:09:50.953 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -2056us[-2034us] +/- 52ms 2023-07-20T19:09:50.953 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1688us[+1688us] +/- 66ms 2023-07-20T19:09:50.953 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T19:09:50.955 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T19:09:50.968 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T19:09:50.969 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T19:09:50.982 DEBUG:teuthology.orchestra.run.smithi061:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T19:09:51.088 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T19:09:51.187 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T19:09:51.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T19:09:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:09:51.332 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T19:09:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:09:51.367 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T19:09:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:09:51.401 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T19:09:51.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T19:09:51.429 INFO:teuthology.task.internal:Duration was 1955.416100 seconds 2023-07-20T19:09:51.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T19:09:51.441 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T19:09:51.442 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T19:09:51.445 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T19:09:51.447 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T19:09:51.494 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T19:09:51.495 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T19:09:51.496 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T19:09:51.820 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T19:09:51.820 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2023-07-20T19:09:51.821 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T19:09:51.848 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2023-07-20T19:09:51.848 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T19:09:51.871 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2023-07-20T19:09:51.871 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T19:09:51.893 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T19:09:51.894 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T19:09:51.896 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T19:09:51.914 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T19:09:51.980 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T19:09:51.981 DEBUG:teuthology.orchestra.run.smithi061:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T19:09:51.984 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T19:09:51.986 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T19:09:52.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T19:09:52.103 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T19:09:52.103 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T19:09:52.132 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T19:09:52.159 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T19:09:52.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T19:09:52.200 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:09:52.203 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:09:52.205 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:09:52.227 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2023-07-20T19:09:52.229 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2023-07-20T19:09:52.253 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2023-07-20T19:09:52.295 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:09:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:09:52.313 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:09:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:09:52.330 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T19:09:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T19:09:52.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T19:09:52.371 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T19:09:52.373 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi061 2023-07-20T19:09:52.373 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T19:09:52.416 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi149 2023-07-20T19:09:52.417 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T19:09:52.461 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/remote/smithi156 2023-07-20T19:09:52.461 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T19:09:52.506 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T19:09:52.507 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T19:09:52.509 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T19:09:52.511 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T19:09:52.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T19:09:52.568 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T19:09:52.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T19:09:52.584 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T19:09:52.585 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T19:09:52.587 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T19:09:52.589 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T19:09:52.602 INFO:teuthology.orchestra.run.smithi061.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 19:09 /home/ubuntu/cephtest 2023-07-20T19:09:52.604 INFO:teuthology.orchestra.run.smithi149.stdout: 262181 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 19:09 /home/ubuntu/cephtest 2023-07-20T19:09:52.608 INFO:teuthology.orchestra.run.smithi156.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 19:09 /home/ubuntu/cephtest 2023-07-20T19:09:52.610 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T19:09:52.620 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T19:09:52.642 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T19:09:57.680 DEBUG:teuthology.nuke:shortname: smithi061 2023-07-20T19:09:57.681 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T19:10:07.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:32:04.780941', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T19:10:11.001 DEBUG:teuthology.nuke:shortname: smithi149 2023-07-20T19:10:11.002 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T19:10:21.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:32:04.779819', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T19:10:21.540 DEBUG:teuthology.nuke:shortname: smithi156 2023-07-20T19:10:21.540 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T19:10:21.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 18:32:04.778471', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T19:10:21.581 INFO:teuthology.orchestra.console:Power off smithi061 2023-07-20T19:10:21.582 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T19:10:21.608 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:21.633 INFO:teuthology.orchestra.console:Power off smithi149 2023-07-20T19:10:21.634 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T19:10:21.662 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:21.693 INFO:teuthology.orchestra.console:Power off smithi156 2023-07-20T19:10:21.696 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T19:10:21.727 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:25.655 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:25.711 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:25.754 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:30.769 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:30.824 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:30.870 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:35.884 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:35.936 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:35.983 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:40.996 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:41.048 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:41.095 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:46.110 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:46.161 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:46.207 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:51.225 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:51.276 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:51.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:56.339 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:56.389 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:10:56.435 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:01.456 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:01.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:01.549 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:06.569 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:06.617 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:06.661 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:06.784 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2023-07-20T19:11:06.877 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2023-07-20T19:11:11.733 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:16.848 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:21.964 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:27.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:32.196 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T19:11:33.312 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-20T19:11:33.313 ERROR:teuthology.orchestra.console:Failed to power off smithi149 2023-07-20T19:11:38.329 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 1955.4160995483398 failure_reason: 'Command failed on smithi061 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5aea0705bbf84ad9b39b50b522198299 status: fail success: false 2023-07-20T19:11:38.330 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T19:11:42.718 INFO:teuthology.run:FAIL